Data storage system having crossbar switch with multi-staged routing

ABSTRACT

A memory system having a backplane with a plurality of receiving slots. Each one of the slots has electrical contacts for providing an indication of Such one of the slots. Each one of the slots has a different slot indication. A plurality of memory boards is provided. Each one of the memory boards is plugged into a corresponding one of the slots. Each one of such boards is coupled to the electrical contacts in the corresponding one of the slots to provide a slot signal indicative of the slot indication provided by the electrical contacts. Each one of such boards has: a memory array region; and a switching network for transferring information between a port of the switching network and a memory on such memory boards The transfer is initiated by a director coupled to such port. The director designates a selected one of the plurality of memory boards. The director provides to the switching network a “tag” indicating such designated one of one of the plurality of memory boards having the memory involved in the requested transfer. The switch network includes a memory board checker, for comparing the slot signal with the “tag” for indicating whether the memory board receiving the “tag” is the director designated one of the plurality of memory boards.

TECHNICAL FIELD

This invention relates generally to data storage systems, and moreparticularly to data storage systems having redundancy arrangements toprotect against total system failure in the event of a failure in acomponent or subassembly of the storage system.

BACKGROUND

As is known in the art, large host computers and servers (collectivelyreferred to herein as “host computer/servers”) require large capacitydata storage systems. These large computer/servers generally includesdata processors, which perform many operations on data introduced to thehost computer/server through peripherals including the data storagesystem. The results of these operations are output to peripherals,including the storage system.

One type of data storage system is a magnetic disk storage system. Herea bank of disk drives and the host computer/server are coupled togetherthrough an interface. The interface includes “front end” or hostcomputer/server controllers (or directors) and “back-end” or diskcontrollers (or directors). The interface operates the controllers (ordirectors) in such a way that they are transparent to the hostcomputer/server. That is, data is stored in, and retrieved from, thebank of disk drives in such a way that the host computer/server merelythinks it is operating with its own local disk drive. One such system isdescribed in U.S. Pat. No. 5,206,939, entitled “System and Method forDisk Mapping and Data Retrieval”, inventors Moshe Yanai, NatanVishlitzky, Bruno Alterescu and Daniel Castel, issued Apr. 27, 1993, andassigned to the same assignee as the present invention.

As described in such U.S. Patent, the interface may also include, inaddition to the host computer/server controllers (or directors) and diskcontrollers (or directors), addressable cache memories. The cache memoryis a semiconductor memory and is provided to rapidly store data from thehost computer/server before storage in the disk drives, and, on theother hand, store data from the disk drives prior to being sent to thehost computer/server. The cache memory being a semiconductor memory, asdistinguished from a magnetic memory as in the case of the disk drives,is much faster than the disk drives in reading and writing data.

The host computer/server controllers, disk controllers and cache memoryare interconnected through a backplane printed circuit board. Moreparticularly, disk controllers are mounted on disk controller printedcircuit boards. The host computer/server controllers are mounted on hostcomputer/server controller printed circuit boards. And, cache memoriesare mounted on cache memory printed circuit boards. The disk directors,host computer/server directors, and cache memory printed circuit boardsplug into the backplane printed circuit board. In order to provide dataintegrity in case of a failure in a director, the backplane printedcircuit board has a pair of buses. One set the disk directors isconnected to one bus and another set of the disk directors is connectedto the other bus. Likewise, one set the host computer/server directorsis connected to one bus and another set of the host computer/serverdirectors is directors connected to the other bus. The cache memoriesare connected to both buses. Each one of the buses provides data,address and control information.

The arrangement is shown schematically in FIG. 1. Thus, the use of twobuses B1, B2 provides a degree of redundancy to protect against a totalsystem failure in the event that the controllers or disk drivesconnected to one bus, fail. Further, the use of two buses increases thedata transfer bandwidth of the system compared to a system having asingle bus. Thus, in operation, when the host computer/server 12 wishesto store data, the host computer 12 issues a write request to one of thefront-end directors 14 (i.e., host computer/server directors) to performa write command. One of the front-end directors 14 replies to therequest and asks the host computer 12 for the data. After the requesthas passed to the requesting one of the front-end directors 14, thedirector 14 determines the size of the data and reserves space in thecache memory 18 to store the request. The front-end director 14 thenproduces control signals on one of the address memory busses B1, B2connected to such front-end director 14 to enable the transfer to thecache memory 18. The host computer/server 12 then transfers the data tothe front-end director 14. The front-end director 14 then advises thehost computer/server 12 that the transfer is complete. The front-enddirector 14 looks up in a Table, not shown, stored in the cache memory18 to determine which one of the back-end directors 20 (i.e., diskdirectors) is to handle this request. The Table maps the hostcomputer/server 12 addresses into an address in the bank 14 of diskdrives. The front-end director 14 then puts a notification in a “mailbox” (not shown and stored in the cache memory 18) for the back-enddirector 20, which is to handle the request, the amount of the data andthe disk address for the data. Other back-end directors 20 poll thecache memory 18 when they are idle to check their “mail boxes”. If thepolled “mail box” indicates a transfer is to be made, the back-enddirector 20 processes the request, addresses the disk drive in the bank22, reads the data from the cache memory 18 and writes it into theaddresses of a disk drive in the bank 22.

When data is to be read from a disk drive in bank 22 to the hostcomputer/server 12 the system operates in a reciprocal manner. Moreparticularly, during a read operation, a read request is instituted bythe host computer/server 12 for data at specified memory locations(i.e., a requested data block). One of the front-end directors 14receives the read request and examines the cache memory 18 to determinewhether the requested data block is stored in the cache memory 18. Ifthe requested data block is in the cache memory 18, the requested datablock is read from the cache memory 18 and is sent to the hostcomputer/server 12. If the front-end director 14 determines that therequested data block is not in the cache memory 18 (i.e., a so-called“cache miss”) and the director 14 writes a note in the cache memory 18(i.e., the “mail box”) that it needs to receive the requested datablock. The back-end directors 20 poll the cache memory 18 to determinewhether there is an action to be taken (i.e., a read operation of therequested block of data). The one of the back-end directors 20 whichpoll the cache memory 18 mail box and detects a read operation reads therequested data block and initiates storage of such requested data blockstored in the cache memory 18. When the storage is completely writteninto the cache memory 18, a read complete indication is placed in the“mail box” in the cache memory 18. It is to be noted that the front-enddirectors 14 are polling the cache memory 18 for read completeindications. When one of the polling front-end directors 14 detects aread complete indication, such front-end director 14 completes thetransfer of the requested data which is now stored in the cache memory18 to the host computer/server 12.

The use of mailboxes and polling requires time to transfer data betweenthe host computer/server 12 and the bank 22 of disk drives thus reducingthe operating bandwidth of the interface.

SUMMARY

In accordance with the present invention, a memory system is providedhaving a backplane with a plurality of receiving slots. Each one of theslots has electrical contacts for providing an indication of such one ofthe slots. Each one of the slots has a different slot indication. Aplurality of memory boards is provided. Each one of the memory boards isplugged into a corresponding one of the slots. Each one of such boardsis coupled to the electrical contacts in the corresponding one of theslots to provide a slot signal indicative of the slot indicationprovided by the electrical contacts. Each one of such boards has: amemory array region; and a switching network for transferringinformation between a port of the switching network and a memory on suchmemory board. The transfer is initiated by a director coupled to suchport. The director designates a selected one of the plurality of memoryboards. The director provides to the switching network a “tag”indicating such designated one of one of the plurality of memory boardshaving the memory involved in the requested transfer. The switch networkincludes a memory board checker, for comparing the slot signal with the“tag” for indicating whether the memory board receiving the “tag,” isthe director designated one of the plurality of memory boards.

In one embodiment, the memory board checker sends an indication to therequesting director slot signal as to whether the memory board receivingthe “tag” is the designated one of the memory boards.

In one embodiment, the switching network includes a decoder responsiveto the indication provided by the memory board checker for inhibitingtransfer of the information at the port to the memory if the memoryboard receiving the “tag” is different from the designated one of thememory boards.

In one embodiment, the switching network includes a decoder responsiveto the indication provided by the memory board checker for inhibitingtransfer of the information at the port to the memory if the memoryboard receiving the “tag” is different from the designated one of thememory boards.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

These and other features of the invention will become more readilyapparent from the following detailed description when read together withthe accompanying drawings, in which:

FIG. 1 is a block diagram of a data storage system according to thePRIOR ART;

FIG. 2 is a block diagram of a data storage system according to theinvention;

FIG. 2A shows the fields of a descriptor used in the system interface ofthe data storage system of FIG. 2;

FIG. 2B shows the filed used in a MAC packet used in the systeminterface of the data storage system of FIG. 2;

FIG. 3 is a sketch of an electrical cabinet storing a system interfaceused in the data storage system of FIG. 2;

FIG. 4 is a diagramatical, isometric sketch showing printed circuitboards providing the system interface of the data storage system of FIG.2;

FIG. 5 is a block diagram of the system interface used in the datastorage system of FIG. 2;

FIG. 6 shows the relationship between FIGS. 6A and 6B which when takentogether is a block diagram showing the connections between front-endand back-end directors to one of a pair of message network boards usedin the system interface of the data storage system of FIG. 2;

FIG. 7 is a block diagram of an exemplary one of the director boardsused in the system interface of he data storage system of FIG. 2;

FIG. 8 is a block diagram of the system interface used in the datastorage system of FIG. 2;

FIG. 8A is a diagram of an exemplary global cache memory board used inthe system interface of FIG. 8;

FIG. 8B is a diagram showing a pair of director boards coupled between apair of host processors and global cache memory boards used in thesystem interface of FIG. 8;

FIGS. 9A, 9B and 9C are a more detailed block diagram of the exemplarycache memory board of FIG. 8A;

FIG. 10 is a block diagram of a crossbar switch used in the memory boardof FIGS. 9A, 9B and 9C;

FIGS. 11A, 11B, 11C and 11D are a block diagram of an upper portinterface section used in the crossbar switch of FIG. 10;

FIGS. 12A, 12B, 12C and 12D are a block diagram of a lower portinterface section used in the crossbar switch of FIG. 10;

FIGS. 13A, 13B, 13C, 13D and 13E are a block diagram of a pair of logicsections used in the memory board of FIGS. 9A, 9B and 9C;

FIGS. 14A, 14B, 14C and 14D are a block diagram of a pair of portcontrollers used in the pair of logic sections of FIGS. 13A, 13B, 13C,13D and 13E;

FIGS. 15A, 15B, 15C, 15D and 15E are a block diagram of a pair ofarbitration logics used in the pair of logic sections of FIGS. 13A, 13B,13C, 13D and 13E and of a watchdog section used for such pair of logicsections;

FIG. 16 is a diagram showing words that make up exemplary informationcycle used in the memory board of FIGS. 9A, 9B and 9C;

FIG. 17 is a Truth Table for a majority gate used in the memory board ofFIGS. 9A, 9B and 9C;

FIG. 18 is a block diagram shown interconnections between one of thearbitration units used in one of the pair of port controllers of FIGS.13A, 13B, 13C, 13D and 13E and a filter used in the arbitration unit ofthe other one of such pair of controllers of FIGS. 13A, 13B, 13C, 13Dand 13E;

FIG. 19 is a timing diagram of signals in arbitration units of FIG. 18used of one of the pair of port controllers of FIGS. 14A, 14B, 14C and14D and a filter used in the arbitration unit used in the other one ofsuch pair of controllers of FIGS. 14A, 14B, 14C and 14D; and

FIGS. 20A, 20B and 20C are a more detailed block diagram of arbitrationsused in the arbitration logics of FIGS. 15A, 15B, 15C, 15D and 15E.

DETAILED DESCRIPTION

Referring now to FIG. 2, a data storage system 100 is shown fortransferring data between a host computer/server 120 and a bank of diskdrives 140 through a system interface 160. The system interface 160includes: a plurality of, here 32 front-end directors 180 ₁-180 ₃₂coupled to the host computer/server 120 via ports-123 ₃₂; a plurality ofback-end directors 200 ₁-200 ₃₂ coupled to the bank of disk drives 140via ports 123 ₃₃-123 ₆₄; a data transfer section 240, having a globalcache memory 220, coupled to the plurality of front-end directors 180₁-180 ₁₆ and the back-end directors 200 ₁-200 ₁₆; and a messagingnetwork 260, operative independently of the data transfer section 240,coupled to the plurality of front-end directors 180 ₁-180 ₃₂ and theplurality of back-end directors 200 ₁-200 ₃₂, as shown. The front-endand back-end directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ are functionallysimilar and include a microprocessor (μP) 299 (i.e., a centralprocessing unit (CPU) and RAM), a message engine/CPU controller 314 anda data pipe 316 to be described in detail in connection with FIGS. 5, 6and 7. Suffice it to say here, however, that the front-end and back-enddirectors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ control data transfer between thehost computer/server 120 and the bank of disk drives 140 in response tomessages passing between the directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂through the messaging network 260. The messages facilitate the datatransfer between host computer/server 120 and the bank of disk drives140 with such data passing through the global cache memory 220 via thedata transfer section 240. More particularly, in the case of thefront-end directors 180 ₁-180 ₃₂, the data passes between the hostcomputer to the global cache memory 220 through the data pipe 316 in thefront-end directors 180 ₁-180 ₃₂ and the messages pass through themessage engine/CPU controller 314 in such front-end directors 180 ₁-180₃₂. In the case of the back-end directors 200 ₁-200 ₃₂ the data passesbetween the back-end directors 200 ₁-200 ₃₂ and the bank of disk drives140 and the global cache memory 220 through the data pipe 316 in theback-end directors 200 ₁-200 ₃₂ and again the messages pass through themessage engine/CPU controller 314 in such back-end director 200 ₁-200₃₂.

With such an arrangement, the cache memory 220 in the data transfersection 240 is not burdened with the task of transferring the directormessaging. Rather the messaging network 260 operates independent of thedata transfer section 240 thereby increasing the operating bandwidth ofthe system interface 160.

In operation, and considering first a read request by the hostcomputer/server 120 (i.e., the host computer/server 120 requests datafrom the bank of disk drives 140), the request is passed from one of aplurality of, here 32, host computer processors 121 ₁-121 ₃₂ in the hostcomputer 120 to one or more of the pair of the front-end directors 180₁-180 ₃₂ connected to such host computer processor 121 ₁-121 ₃₂. (It isnoted that in the host computer 120, each one of the host computerprocessors 121 ₁-121 ₃₂ is coupled to here a pair (but not limited to apair) of the front-end directors 180 ₁-180 ₃₂, to provide redundancy inthe event of a failure in one of the front end-directors 181 ₁-181 ₃₂coupled thereto. Likewise, the bank of disk drives 140 has a pluralityof, here 32, disk drives 141 ₁-141 ₃₂, each disk drive 141 ₁-141 ₃₂being coupled to here a pair (but not limited to a pair) of the back-enddirectors 200 ₁-200 ₃₂, to provide redundancy in the event of a failurein one of the back-end directors 200 ₁-200 ₃₂ coupled thereto). Eachfront-end director 180 ₁-180 ₃₂ includes a microprocessor (μP) 299(i.e., a central processing unit (CPU) and RAM) and will be described indetail in connection with FIGS. 5 and 7. Suffice it to say here,however, that the microprocessor 299 makes a request for the data fromthe global cache memory 220. The global cache memory 220 has a residentcache management table, not shown. Every director 180 ₁-180 ₃₂, 200₁-200 ₃₂ has access to the resident cache management table and everytime a front-end director 180 ₁-180 ₃₂ requests a data transfer, thefront-end director 180 ₁-180 ₃₂ must query the global cache memory 220to determine whether the requested data is in the global cache memory220. If the requested data is in the global cache memory 220 (i.e., aread “hit”), the front-end director 180 ₁-180 ₃₂, more particularly themicroprocessor 299 therein, mediates a DMA (Direct Memory Access)operation for the global cache memory 220 and the requested data istransferred to the requesting host computer processor 121 ₁-121 ₃₂.

If, on the other hand, the front-end director 180 ₁-180 ₃₂ receiving thedata request determines that the requested data is not in the globalcache memory 220 (i.e., a “miss”) as a result of a query of the cachemanagement table in the global cache memory 220, such front-end director180 ₁-180 ₃₂ concludes that the requested data is in the bank of diskdrives 140. Thus the front-end director 180 ₁-180 ₃₂ that received therequest for the data must make a request for the data from one of theback-end directors 200 ₁-200 ₃₂ in order for such back-end director 200₁-200 ₃₂ to request the data from the bank of disk drives 140. Themapping of which back-end directors 200 ₁-200 ₃₂ control which diskdrives 141 ₁-141 ₃₂ in the bank of disk drives 140 is determined duringa power-up initialization phase. The map is stored in the global cachememory 220. Thus, when the front-end director 180 ₁-180 ₃₂ makes arequest for data from the global cache memory 220 and determines thatthe requested data is not in the global cache memory 220 (i.e., a“miss”), the front-end director 180 ₁-180 ₃₂ is also advised by the mapin the global cache memory 220 of the back-end director 200 ₁-200 ₃₂responsible for the requested data in the bank of disk drives 140. Therequesting front-end director 180 ₁-180 ₃₂ then must make a request forthe data in the bank of disk drives 140 from the map designated back-enddirector 200 ₁-200 ₃₂. This request between the front-end director 180₁-180 ₃₂ and the appropriate one of the back-end directors 200 ₁-200 ₃₂(as determined by the map stored in the global cache memory 200) is by amessage which passes from the front-end director 180 ₁-180 ₃₂ throughthe message network 260 to the appropriate back-end director 200 ₁-200₃₂. It is noted then that the message does not pass through the globalcache memory 220 (i.e., does not pass through the data transfer section240) but rather passes through the separate, independent message network260. Thus, communication between the directors 180 ₁-180 ₃₂, 200 ₁-200₃₂ is through the message network 260 and not through the global cachememory 220. Consequently, valuable bandwidth for the global cache memory220 is not used for messaging among the directors 180 ₁-180 ₃₂, 200₁-200 ₃₂.

Thus, on a global cache memory 220 “read miss”, the front-end director180 ₁-180 ₃₂ sends a message to the appropriate one of the back-enddirectors 200 ₁-200 ₃₂ through the message network 260 to instruct suchback-end director 200 ₁-200 ₃₂ to transfer the requested data from thebank of disk drives 140 to the global cache memory 220. Whenaccomplished, the back-end director 200 ₁-200 ₃₂ advises the requestingfront-end director 180 ₁-180 ₃₂ that the transfer is accomplished by amessage, which passes from the back-end director 200 ₁-200 ₃₂ to thefront-end director 180 ₁-180 ₃₂ through the message network 260. Inresponse to the acknowledgement signal, the front-end director 180 ₁-180₃₂ is thereby advised that such front-end director 180 ₁-180 ₃₂ cantransfer the data from the global cache memory 220 to the requestinghost computer processor 121 ₁-121 ₃₂ as described above when there is acache “read hit”.

It should be noted that there might be one or more back-end directors200 ₁-200 ₃₂ responsible for the requested data. Thus, if only oneback-end director 200 ₁-200 ₃₂ is responsible for the requested data,the requesting front-end director 180 ₁-180 ₃₂ sends a uni-cast messagevia the message network 260 to only that specific one of the back-enddirectors 200 ₁-200 ₃₂. On the other hand, if more than one of theback-end directors 200 ₁-200 ₃₂ is responsible for the requested data, amulti-cast message (here implemented as a series of uni-cast messages)is sent by the requesting one of the front-end directors 180 ₁-180 ₃₂ toall of the back-end directors 200 ₁-200 ₃₂ having responsibility for therequested data. In any event, with both a uni-cast or multi-castmessage, such message is passed through the message network 260 and notthrough the data transfer section 240 (i.e., not through the globalcache memory 220).

Likewise, it should be noted that while one of the host computerprocessors 121 ₁-121 ₃₂ might request data, the acknowledgement signalmay be sent to the requesting host computer processor 121 ₁ or one ormore other host computer processors 121 ₁-121 ₃₂ via a multi-cast (i.e.,sequence of uni-cast) messages through the message network 260 tocomplete the data read operation.

Considering a write operation, the host computer 120 wishes to writedata into storage (i.e., into the bank of disk drives 140). One of thefront-end directors 180 ₁-180 ₃₂ receives the data from the hostcomputer 120 and writes it into the global cache memory 220. Thefront-end director 180 ₁-180 ₃₂ then requests the transfer of such dataafter some period of time when the back-end director 200 ₁-200 ₃₂determines that the data can be removed from such cache memory 220 andstored in the bank of disk drives 140. Before the transfer to the bankof disk drives 140, the data in the cache memory 220 is tagged with abit as “fresh data” (i.e., data which has not been transferred to thebank of disk drives 140, that is data which is “write pending”). Thus,if there are multiple write requests for the same memory location in theglobal cache memory 220 (e.g., a particular bank account) before beingtransferred to the bank of disk drives 140, the data is overwritten inthe cache memory 220 with the most recent data. Each time data istransferred to the global cache memory 220, the front-end director 180₁-180 ₃₂ controlling the transfer also informs the host computer 120that the transfer is complete to thereby free-up the host computer 120for other data transfers.

When it is time to transfer the data in the global cache memory 220 tothe bank of disk drives 140, as determined by the back-end director 200₁-200 ₃₂, the back-end director 200 ₁-200 ₃₂ transfers the data from theglobal cache memory 220 to the bank of disk drives 140 and resets thetag associated with data in the global cache memory 220 (i.e., un-tagsthe data) to indicate that the data in the global cache memory 220 hasbeen transferred to the bank of disk drives 140. It is noted that theun-tagged data in the global cache memory 220 remains there untiloverwritten with new data.

Referring now to FIGS. 3 and 4, the system interface 160 is shown toinclude an electrical cabinet 300 having stored therein: a plurality of,here eight front-end director boards 190 ₁-190 ₈, each one having herefour of the front-end directors 180 ₁-180 ₃₂; a plurality of, here eightback-end director boards 210 ₁-210 ₈, each one having here four of theback-end directors 200 ₁-200 ₃₂; and a plurality of, here eight, memoryboards 220 ′ which together make up the global cache memory 220. Theseboards plug into the front side of a backplane 302. (It is noted thatthe backplane 302 is a mid-plane printed circuit board). Plugged intothe backside of the backplane 302 are message network boards 304 ₁, 304₂. The backside of the backplane 302 has plugged into it adapter boards,not shown in FIGS. 2-4, which couple the boards plugged into theback-side of the backplane 302 with the computer 120 and the bank ofdisk drives 140 as shown in FIG. 2. That is, referring again briefly toFIG. 2, an I/O adapter, not shown, is coupled between each one of thefront-end directors 180 ₁-180 ₃₂ and the host computer 120 and an I/Oadapter, not shown, is coupled between each one of the back-enddirectors 200 ₁-200 ₃₂ and the bank of disk drives 140.

Referring now to FIG. 5, the system interface 160 is shown to includethe director boards 190 ₁-190 ₈, 210 ₁-210 ₈ and the global cache memory220, plugged into the backplane 302 and the disk drives 141 ₁-141 ₃₂ inthe bank of disk drives along with the host computer 120 also pluggedinto the backplane 302 via I/O adapter boards, not shown. The messagenetwork 260 (FIG. 2) includes the message network boards 304 ₁ and 304₂. Each one of the message network boards 304 ₁ and 304 ₂ is identicalin construction. A pair of message network boards 304 ₁ and 304 ₂ isused for redundancy and for message load balancing. Thus, each messagenetwork board 304 ₁, 304 ₂, includes a controller 306, (i.e., aninitialization and diagnostic processor comprising a CPU, systemcontroller interface and memory, as shown in FIG. 6 for one of themessage network boards 304 ₁, 304 ₂, here board 304 ₁) and a crossbarswitch section 308 (e.g., a switching fabric made up of here fourswitches 308 ₁-308 ₄).

Referring again to FIG. 5, each one of the director boards 190 ₁-210 ₈includes, as noted above four of the directors 180 ₁-180 ₃₂, 200 ₁-200₃₂ (FIG. 2). It is noted that the director boards 190 ₁-190 ₈ havingfour front-end directors per board, 180 ₁-180 ₃₂ are referred to asfront-end directors, and the director boards 210 ₁-210 ₈ having fourback-end directors per board, 200 ₁-200 ₃₂ are referred to as back-enddirectors. Each one of the directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ includesa CPU 310, a RAM 312 (which make up the microprocessor 299 referred toabove), the message engine/CPU controller 314, and the data pipe 316.

Each one of the director boards 190 ₁-210 ₈ includes a crossbar switch318. The crossbar switch 318 has four input/output ports 319, each onebeing coupled to the data pipe 316 of a corresponding one of the fourdirectors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ on the director board. 190 ₁-210 ₈.The crossbar switch 318 has eight output/input ports collectivelyidentified in FIG. 5 by numerical designation 321 (which plug into thebackplane 302. The crossbar switch 318 on the front-end director boards191 ₁-191 ₈ is used for coupling the data pipe 316 of a selected one ofthe four front-end directors 180 ₁-180 ₃₂ on the front-end directorboard 190 ₁-190 ₈ to the global cache memory 220 via the backplane 302and I/O adapter, not shown. The crossbar switch 318 on the back-enddirector boards 210 ₁-210 ₈ is used for coupling the data pipe 316 of aselected one of the four back-end directors 200 ₁-200 ₃₂ on the back-enddirector board 210 ₁-210 ₈ to the global cache memory 220 via thebackplane 302 and I/O adapter, not shown. Thus, referring to FIG. 2, thedata pipe 316 in the front-end directors 180 ₁-180 ₃₂ couples databetween the host computer 120 and the global cache memory 220 while thedata pipe 316 in the back-end directors 200 ₁-200 ₃₂ couples databetween the bank of disk drives 140 and the global cache memory 220. Itis noted that there are separate point-to-point data paths P₁-P₆₄ (FIG.2) between each one of the directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ and theglobal cache memory 220. It is also noted that the backplane 302 is apassive backplane because it is made up of only etched conductors on oneor more layers to a printed circuit board. That is, the backplane 302does not have any active components.

Referring again to FIG. 5, each one of the director boards 190 ₁-210 ₈includes a crossbar switch 320. Each crossbar switch 320 has fourinput/output ports 323, each one of the four input/output ports 323being coupled to the message engine/CPU controller 314 of acorresponding one of the four directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ onthe director board 190 ₁-210 ₈. Each crossbar switch 320 has a pair ofoutput/input ports 325 ₁, 325 ₂, which plug into the backplane 302. Eachport 325 ₁-325 ₂ is coupled to a corresponding one of the messagenetwork boards 304 ₁, 304 ₂, respectively, through the backplane 302.The crossbar switch 320 on the front-end director boards 190 ₁-190 ₈ isused to couple the messages between the message engine/CPU controller314 of a selected one of the four front-end directors 180 ₁-180 ₃₂ onthe front-end director boards 190 ₁-190 ₈ and the message network 260,FIG. 2. Likewise, the back-end director boards 210 ₁-210 ₈ are used tocouple the messages produced by a selected one of the four back-enddirectors 200 ₁-200 ₃₂ on the back-end director board 210 ₁-210 ₈between the message engine/CPU controller 314 of a selected one of suchfour back-end directors and the message network 260 (FIG. 2). Thus,referring also to FIG. 2, instead of having a separate dedicated messagepath between each one of the directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ andthe message network 260 (which would require M individual connections tothe backplane 302 for each of the directors, where M is an integer),here only M/4 individual connections are required). Thus, the totalnumber of connections between the directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂and the backplane 302 is reduced to ¼th. Thus, it Should be noted fromFIGS. 2 and 5 that the message network 260 (FIG. 2) includes thecrossbar switch 320 and the message network boards 304 ₁, 304 ₂.

Each message is a 64-byte descriptor, shown in FIG. 2A, which is createdby the CPU 310 (FIG. 5) under software control and is stored in a sendqueue in RAM 312. When the message is to be read from the send queue inRAM 312 and transmitted through the message network 260 (FIG. 2) to oneor more other directors via a DMA operation to be described, it ispacketized in the packetizer portion of packetizer/de-packetizer 428(FIG. 7) into a MAC type packet, shown in FIG. 2B, here using the NGIOprotocol specification. There are three types of packets: a messagepacket section; an acknowledgement packet; and a message network fabricmanagement packet, the latter being used to establish the messagenetwork routing during initialization (i.e., during power-up). Each oneof the MAC packets has: an 8-byte header which includes source (i.e.,transmitting director) and destination (i.e., receiving director)address; a payload; and terminates with a 4-byte Cyclic Redundancy Check(CRC), as shown in FIG. 2B. The acknowledgement packet (i.e., signal)has a 4-byte acknowledgment payload section. The message packet has a32-byte payload section. The Fabric Management Packet (FMP) has a256-byte payload section. The MAC packet is sent to the crossbar switch320. The destination portion of the packet is used to indicate thedestination for the message and is decoded by the switch 320 todetermine which port the message is to be routed. The decoding processuses a decoder table 327 in the switch 318, such table being initializedby controller during power-up by the initialization and diagnosticprocessor (controller) 306 (FIG. 5). The table 327 (FIG. 7) provides therelationship between the destination address portion of the MAC packet,which identifies the routing for the message and the one of the fourdirectors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ on the director board 190 ₁-190 ₈,210 ₁-210 ₈ or to one of the message network boards 304 ₁, 304 ₂ towhich the message is to be directed.

More particularly, and referring to FIG. 5, a pair of output/input ports325 ₁, 325 ₂ is provided for each one of the crossbar switches 320, eachone being coupled to a corresponding one of the pair of message networkboards 304 ₁, 304 ₂. Thus, each one of the message network boards 304 ₁,304 ₂ has sixteen input/output ports 322 ₁-322 ₁₆, each one beingcoupled to a corresponding one of the output/input ports 325 ₁, 325 ₂,respectively, of a corresponding one of the director boards 190 ₁-190 ₈,210 ₁-210 ₈ through the backplane 302, as shown. Thus, consideringexemplary message network board 304 ₁, FIG. 6, each switch 308 ₁-308 ₄also includes three coupling ports 324 ₁-324 ₃. The coupling ports 324₁-324 ₃ are used to interconnect the switches 322 ₁-322 ₄, as shown inFIG. 6. Thus, considering message network board 304 ₁, input/outputports 322 ₁-322 ₈ are coupled to output/input ports 325 ₁ of front-enddirector boards 190 ₁-190 ₈ and input/output ports 322 ₉-322 ₁₆ arecoupled to output/input ports 325 ₁ of back-end director boards 210₁-210 ₈, as shown. Likewise, considering message network board 304 ₂,input/output ports 322 ₁-322 ₈ thereof are coupled, via the backplane302, to output/input ports 325 ₂ of front-end director boards 190 ₁-190₈ and input/output ports 322 ₉-322 ₁₆ are coupled, via the backplane302, to output/input ports 325 ₂ of back-end director boards 210 ₁-210₈.

As noted above, each one of the message network boards 304 ₁, 304 ₂includes a processor 306 (FIG. 5) and a crossbar switch section 308having four switches 308 ₁-308 ₄, as shown in FIGS. 5 and 6. Theswitches 308 ₁-308 ₄ are interconnected as shown so that messages canpass between any pair of the input/output ports 322 ₁-322 ₁₆. Thus, itfollow that a message from any one of the front-end directors 180 ₁-180₃₂ can be coupled to another one of the front-end directors 180 ₁-180 ₃₂and/or to any one of the back-end directors 200 ₁-200 ₃₂. Likewise, amessage from any one of the back-end directors 180 ₁-180 ₃₂ can becoupled to another one of the back-end directors 180 ₁-180 ₃₂ and/or.toany one of the front-end directors 200 ₁-200 ₃₂.

As noted above, each MAC packet (FIG. 2B) includes in an addressdestination portion and a data payload portion. The MAC header is usedto indicate the destination for the MAC packet and such MAC header isdecoded by the switch to determine which port the MAC packet is to berouted. The decoding process uses a table in the switch 308 ₁-308 ₄,such table being initialized by processor 306 during power-up. The tableprovides the relationship between the MAC header, which identifies thedestination for the MAC packet and the route to be taken through themessage network. Thus, after initialization, the switches 320 and theswitches 308 ₁-308 ₄ in switch section 308 provides packet routing whichenables each one of the directors 180 ₁-180 ₃₂, 200 ₁-200 ₃₂ to transmita message between itself and any other one of the directors, regardlessof whether such other director is on the same director board 190 ₁-190₈, 210 ₁-210 ₈ or on a different director board. Further, the MAC packethas an additional bit B in the header thereof, as shown in FIG. 2B,which enables the message to pass through message network board 304 ₁ orthrough message network board 304 ₂. During normal operation, thisadditional bit B is toggled between a logic 1 and a logic 0 so that onemessage passes through one of the redundant message network boards 304₁, 304 ₂ and the next message to pass through the other one of themessage network boards 304 ₁, 304 ₂ to balance the load requirement onthe system. However, in the event of a failure in one of the messagenetwork boards 304 ₁, 304 ₂, the non-failed one of the boards 304 ₁, 304₂ is used exclusively until the failed message network board isreplaced.

Referring now to FIG. 7, an exemplary one of the director boards 190₁-190 ₈, 210 ₁-210 ₈, here director board 190 ₁ is shown to includedirectors 180 ₁, 180 ₃, 180 ₅ and 180 ₇. An exemplary one of thedirectors 180 ₁-180 ₄, here director 180 ₁ is shown in detail to includethe data pipe 316, the message engine/CPU controller 314, the RAM 312,and the CPU 310 all coupled to the CPU interface bus 317, as shown. Theexemplary director 180 ₁ also includes: a local cache memory 319 (whichis coupled to the CPU 310); the crossbar switch 318; and, the crossbarswitch 320, described briefly above in connection with FIGS. 5 and 6.The data pipe 316 includes a protocol translator 400, a quad port RAM402 and a quad port RAM controller 404 arranged as shown. Briefly, theprotocol translator 400 converts between the protocol of the hostcomputer 120, in the case of a front-end director 180 ₁-180 ₃₂, (andbetween the protocol used by the disk drives in bank 140 in the case ofa back-end director 200 ₁-200 ₃₂) and the protocol between the directors180 ₁-180 ₃, 200 ₁-200 ₃₂ and the global memory 220 (FIG. 2). Moreparticularly, the protocol used the host computer 120 may, for example,be fiber channel, SCSI, ESCON or FICON, for example, as determined bythe manufacture of the host computer 120 while the protocol usedinternal to the system interface 160 (FIG. 2) may be selected by themanufacturer of the interface 160. The quad port RAM 402 is a FIFOcontrolled by controller 404 because the rate data coming into the RAM402 may be different from the rate data leaving the RAM 402. The RAM 402has four ports, each adapted to handle an 18 bit digital word. Here, theprotocol translator 400 produces 36 bit digital words for the systeminterface 160 (FIG. 2) protocol, one 18 bit portion of the word iscoupled to one of a pair of the ports of the quad port RAM 402 and theother 18 bit portion of the word is coupled to the other one of the pairof the ports of the quad port RAM 402. The quad port RAM has a pair ofports 402A, 402B, each one of to ports 402A, 402B being adapted tohandle an 18 bit digital word. Each one of the ports 402A, 402B isindependently controllable and has independent, but arbitrated, accessto the memory array within the RAM 402. Data is transferred between theports 402A, 402B and the cache memory 220 (FIG. 2) through the crossbarswitch 318, as shown.

The crossbar switch 318 includes a pair of switches 406A, 406B. Each oneof the switches 406A, 406B includes four input/output director-sideports D₁-D₄ (collectively referred to above in connection with FIG. 5 asport 319) and four input/output memory-side ports M₁-M₄, M₅-M₈,respectively, as indicated. The input/output memory-side ports M₁-M₄,M₅-M₈ were collectively referred to above in connection with FIG. 5 asport 317). The director-side ports D₁-D₄ of switch 406A are connected tothe 402A ports of the quad port RAMs 402 in each one the directors 180₁, 180 ₃, 180 ₅ and 180 ₇, as indicated. Likewise, director-side portsof switch 406B are connected to the 402B ports of the quad port RAMs 402in each one the directors 180 ₁, 180 ₃, 180 ₅, and 180 ₇, as indicated.The ports D₁-D₄ are selectively coupled to the ports M₁-M₄ in accordancewith control words provided to the switch 406A by the controllers indirectors 180 ₁, 180 ₃, 180 ₅, 180 ₇ on busses R_(A1)-R_(A4),respectively, and the ports D₁-D₄ are coupled to ports M₅-M₈ inaccordance with the control words provided to switch 406B by thecontrollers in directors 180 ₁, 180 ₃, 180 ₅, 180 ₇ on bussesR_(B1)-R_(B4), as indicated. The signals on buses R_(A1)-R_(A4) arerequest signals. Thus, port 402A of any one of the directors 180 ₁, 180₃, 180 ₅, 180 ₇ may be coupled to any one of the ports M₁-M₄ of switch406A, selectively in accordance with the request signals on busesR_(A1)-R_(A4). Likewise, port 402B of any one of the directors 180 ₁-180₄ may be coupled to any one of the ports M₅-M₈ of switch 406B,selectively in accordance with the request signals on busesR_(B1)-R_(B4). The coupling between the director boards 190 ₁-190 ₈, 210₁-210 ₈ and the global cache memory 220 is shown in FIG. 8.

More particularly, and referring also to FIG. 2, as noted above, eachone of the host computer processors 121 ₁-121 ₃₂ in the host computer120 is coupled to a pair of the front-end directors 180 ₁-180 ₃₂, toprovide redundancy in the event of a failure in one of the frontend-directors 181 ₁-181 ₃₂ coupled thereto. Likewise, the bank of diskdrives 140 has a plurality of, here 32, disk drives 141 ₁-141 ₃₂, eachdisk drive 141 ₁-141 ₃₂ being coupled to a pair of the back-enddirectors 200 ₁-200 ₃₂, to provide redundancy in the event of a failurein one of the back-end directors 200 ₁-200 ₃₂ coupled thereto). Thus,considering exemplary host computer processor 121 ₁, such processor 121₁ is coupled to a pair of front-end directors 180 ₁, 180 ₂. Thus, ifdirector 180 ₁ fails, the host computer processor 121 ₁ can still accessthe system interface 160, albeit by the other front-end director 180 ₂.Thus, directors 180 ₁ and 180 ₂ are considered redundancy pairs ofdirectors. Likewise, other redundancy pairs of front-end directors are:front-end directors 180 ₃, 180 ₄; 180 ₅, 180 ₆; 180 ₇, 180 ₈; 180 ₉, 180₁₀; 180 ₁₁, 180 ₁₂; 180 ₁₃, 180 ₁₄; 180 ₁₅, 180 ₁₆; 180 ₁₇, 180 ₁₈; 180₁₉, 180 ₂₀; 180 ₂₁, 180 ₂₂, 180 ₂₃, 180 ₂₄; 180 ₂₅, 180 ₂₆; 180 ₂₇, 180₂₈; 180 ₂₉, 180 ₃₀; and 180 ₃₁, 180 ₃₂ (only directors 180 ₃₁ and 180 ₃₂being shown in FIG. 2).

Likewise, disk drive 141 ₁ is coupled to a pair of back-end directors200 ₁, 200 ₂. Thus, if director 200 ₁ fails, the disk drive 141 ₁ canstill access the system interface 160, albeit by the other back-enddirector 180 ₂. Thus, directors 200 ₁ and 200 ₂ are consideredredundancy pairs of directors. Likewise, other redundancy pairs ofback-end directors are: back-end directors 200 ₃, 200 ₄; 200 ₅, 200 ₆;200 ₇, 200 ₈; 200 ₉, 200 ₁₀; 200 ₁₁, 200 ₁₂; 200 ₁₃, 200 ₁₄; 200 ₁₅, 200₁₆, 200 ₁₇, 200 ₁₈; 200 ₁₉, 200 ₂₀; 200 ₂₁, 200 ₂₂; 200 ₂₃, 200 ₂₄; 200₂₅, 200 ₂₆; 200 ₂₇, 200 ₂₈; 200 ₂₉, 200 ₃₀; and 200 ₃₁, 200 ₃₂ (onlydirectors 200 ₃₁ and 200 ₃₂ being shown in FIG. 2). Further, referringalso to FIG. 8, the global cache memory 220 includes a plurality of,here eight, cache memory boards 220 ₁-220 ₈, as shown. Still further,referring to FIG. 8A, an exemplary one of the cache memory boards, hereboard 220 ₁ is shown in detail and will be described in detail inconnection with FIGS. 23-29. Here, each cache memory board includes fourmemory array regions, an exemplary one thereof being shown and describedin connection with FIG. 6 of U.S. Pat. No. 5,943,287 entitled “FaultTolerant Memory System”, John K. Walton, inventor, issued Aug. 24, 1999and assigned to the same assignee as the present invention, the entiresubject matter therein being incorporated herein by reference. Furtherdetail of the exemplary one of the cache memory boards.

As shown in FIG. 8A, the board 220 ₁ includes a plurality of, here fourRAM memory arrays, each one of the arrays has a pair of redundant ports,i.e., an A port and a B port. The board itself has sixteen ports; a setof eight A ports M_(A1)-M_(A8) and a set of eight B ports M_(B1)-M_(B8).Four of the eight A port, here A ports M_(A1)-M_(A4) are coupled to theM₁ port of each of the front-end director boards 190 ₁, 190 ₃, 190 ₅,and 190 ₇, respectively, as indicated in FIG. 8. Four of the eight Bport, here B ports M_(B1)-M_(B4) are coupled to the M₁ port of each ofthe front-end director boards 190 ₂, 190 ₄, 190 ₆, and 190 ₈,respectively, as indicated in FIG. 8. The other four of the eight Aport, here A ports M_(A5)-M_(A8) are coupled to the M₁ port of each ofthe back-end director boards 210 ₁, 210 ₃, 210 ₅, and 210 ₇,respectively, as indicated in FIG. 8. The other four of the eight Bport, here B ports M_(B5)-M_(B8) are coupled to the M₁ port of each ofthe back-end director boards 210 ₂, 210 ₄, 210 ₆, and 210 ₈,respectively, as indicated in FIG. 8

Considering the exemplary four A ports M_(A1)-M_(A4), each one of thefour A ports M_(A1)-M_(A4) can be coupled to the A port of any one ofthe memory arrays through the logic network 221 _(1A), to be describedin more detail in connection with FIGS. 25, 126 and 27. Thus,considering port M_(A1), such port can be coupled to the A port of thefour memory arrays. Likewise, considering the four A portsM_(A5)-M_(A8), each one of the four A ports M_(A5)-M_(A8) can be coupledto the A port of any one of the memory arrays through the logic network221 _(1B). Likewise, considering the four B ports M_(B1)-M_(B4), eachone of the four B ports M_(B1)-M_(B4) can be coupled to the B port ofany one of the memory arrays through logic network 221 _(1B). Likewise,considering the four B ports M_(B5)-M_(B8), each one of the four B portsM_(B5)-M_(B8) can be coupled to the B port of any one of the memoryarrays through the logic network 221 _(2B). Thus, considering portM_(B1), such port can be coupled to the B port of the four memoryarrays. Thus, there are two paths data and control from either afront-end director 180 ₁-180 ₃₂ or a back-end director 200 ₁-200 ₃₂ canreach each one of the four memory arrays on the memory board. Thus,there are eight sets of redundant ports on a memory board, i.e., portsM_(A1), M_(B1); M_(A2), M_(B2); M_(A3), M_(B3); M_(A4), MB₄; MA₅,M_(B5); M_(A6), M_(B6); M_(A7), M_(B7); and M_(A8), M_(B8). Further, asnoted above each one of the directors has a pair of redundant ports,i.e. a 402A port and a 402B port (FIG. 7). Thus, for each pair ofredundant directors, the A port (i.e., port 402A) of one of thedirectors in the pair is connected to one of the pair of redundantmemory ports and the B port (i.e., 402B) of the other one of thedirectors in such pair is connected to the other one of the pair ofredundant memory ports.

More particularly, referring to FIG. 8B, an exemplary pair of redundantdirectors is shown, here, for example, front-end director 180 ₁ andfront end-director 180 ₂. It is first noted that the directors 180 ₁,180 ₂ in each redundant pair of directors must be on different directorboards, here boards 190 ₁, 190 ₂, respectively. Thus, here front-enddirector boards 190 ₁-190 ₈ have thereon: front-end directors 180 ₁, 180₃, 180 ₅ and 180 ₇; front-end directors 180 ₂, 180 ₄, 180 ₆ and 180 ₈;front end directors 180 ₉, 180 ₁₁, 180 ₁₃ and 180 ₁₅; front enddirectors 180 ₁₀, 180 ₁₂, 180 ₁₄ and 180 ₁₆; front-end directors 180 ₁₇,180 ₁₉, 180 ₂₁ and 180 ₂₃, front-end directors 180 ₁₈, 180 ₂₀, 180 ₂₂and 180 ₂₄; front-end directors 180 ₂₅, 180 ₂₇, 180 ₂₉ and 180 ₃₁;front-end directors 180 ₁₈, 180 ₂₀, 180 ₂₂ and 180 ₂₄. Thus, hereback-end director boards 210 ₁-210 ₈ have thereon: back-end directors200 ₁, 200 ₃, 200 ₅ and 200 ₇; back-end directors 200 ₂, 200 ₄, 200 ₆and 200 ₈; back-end directors 200 ₉, 200 ₁₁, 200 ₁₃ and 200 ₁₅; back-enddirectors 200 ₁₀, 200 ₁₂, 200 ₁₄ and 200 ₁₆; back-end directors 200 ₁₇,200 ₁₉, 200 ₂₁ and 200 ₂₃; back-end directors 200 ₁₈, 200 ₂₀, 200 ₂₂ and200 ₂₄; back-end directors 200 ₂₅, 200 ₂₇, 200 ₂₉ and 200 ₃₁; back-enddirectors 200 ₁₈, 200 ₂₀, 200 ₂₂ and 200 ₂₄.

Thus, here front-end director 180 ₁, shown in FIG. 8A, is on front-enddirector board 190 ₁ and its redundant front-end director 180 ₂, shownin FIG. 8B, is on anther front-end director board, here for example,front-end director board 190 ₂. As described above, the port 402A of thequad port RAM 402 (i.e., the A port referred to above) is connected toswitch 406A of crossbar switch 318 and the port 402B of the quad portRAM 402 (i.e., the B port referred to above) is connected to switch 406Bof crossbar switch 318. Likewise, for redundant director 180 ₂. However,the ports M₁-M₄ of switch 406A of director 180 ₁ are connected to theM_(A1) ports of global cache memory boards 220 ₁-200 ₄, as shown, whilefor its redundancy director 180 ₂, the ports M₁-M₄ of switch 406A areconnected to the redundant M_(B1) ports of global cache memory boards220 ₁-200 ₄, as shown.

Further details are provided in co-pending patent application Ser. No.09/561,531 filed Apr. 28, 2000 and Ser. No. 09/561,161 assigned to thesame assignee as the present patent application, the entire subjectmatter thereof being incorporated herein by reference.

CACHE MEMORY BOARDS

Referring again to FIG. 8, the system includes a plurality of, hereeight, memory boards. As described above in connection with FIG. 8A,each one of the memory boards includes four memory array regions R₁-R₄.Referring now to FIGS. 9A, 9B and 90C, an exemplary one of the cachememory boards in the cache memory 220 (FIG. 8), here cache memory board220 ₁, is shown in more detail to include, here, the four logic networks221 _(1B), 221 _(2B), 221 _(1A), and 221 _(2A) and, here eightinterface, or memory region control, sections, here logic sections 5010₁-5010 ₈, arranged as shown.

Each one of the four logic networks 221 _(1B), 221 _(2B), 221 _(1A), and221 _(2A) includes four sets of serial-to-parallel converters (S/P),each one of the sets having four of the S/P converters. The sets of S/Pconverters are coupled between ports M_(B1)-M_(B4), M_(B5)-M_(B8),M_(A1)-M_(A4), and M_(A5)—M_(A5), respectively, and a corresponding oneof four crossbar switches 5004 ₁-5004 ₄. The S/Ps convert between aserial stream of information (i.e., data, address, and control. CyclicRedundancy Checks (CRCs), signaling semaphores, etc.) at portsM_(B1)-M_(B8), M_(A1)-M_(A8), and a parallel stream of the informationwhich passes through the crossbar switches 5004 ₁-5004 ₄. Thus, here thecrossbar switches 5004 ₁-5004 ₄ process parallel information.Information is transferred between directors and the crossbar switchesas transfers, or information cycles. An exemplary information transferfor information passing for storage in the memory array region is shownin FIG. 16. Each information cycle is shown to include a plurality ofsixteen bit words, each word being associated with a clock pulse. Thus,first word 0 is shown to include protocol signaling (e.g., semaphore)and a terminating “start-frame” indication. The next word 1 includesmemory control information. The next three words, 2-4, include memoryaddress (ADDR) information. The next word, 5, is a “tag” which indicatedthe memory board, memory array region, and other information to bedescribed. The next two words, 6 and 7, provide Cyclic Redundancy Checks(CRC) information regarding the address (ADDR_CRC). The DATA to bewritten into the memory then follows. The number of words of DATA isvariable and here is between 4 words and 256 words. The informationcycle terminates with two words, X and Y which include DATA CRCinformation.

As will be described in more detail below, the cache memory board 220 ₁is a multi-ported design which allows equal access to one of several,here four, regions of memory (i.e., here memory array regions R₁-R₄)from any of here sixteen ports M_(B1)-M_(B8), M_(A1)-M_(A8). The sixteenports M_(B1)-M_(B8), M_(A1)-M_(A8) are grouped into four sets S₁-S₄.Each one of the sets S₁-S₄ is associated with, i.e., coupled to, acorresponding one of the four crossbar switches 5004 ₁-5004 ₄,respectively, as indicated. Each one of the crossbar switches 5004₁-5004 ₄ interconnects its upper foul ports 5006 ₁-5006 ₄ to acorresponding one of the four memory regions R₁-R₄ in a point-to-pointfashion. Thus, between the four crossbar switches 5004 ₁-5004 ₄ and thefour memory regions R₁-R₄ there are sixteen potential uniqueinterconnects.

The communication between any port M_(B1)-M_(B8), M_(A1)-M_(A8) and itscorresponding crossbar switch 5004 ₁-5004 ₄ is protected by CyclicRedundancy Check (CRC) defined by CCITT-V.41. The communication betweena crossbar switch 5004 ₁-5004 ₄ and the memory array region R₁-R₄ isprotected by byte parity (p). There is a pipelined architecture from theport M_(B1)-M_(B8), M_(A1)-M_(A8). Such architecture includes a pipelinehaving the crossbar switches 5004 ₁-5004 ₄, the logic sections 5010₁-5010 ₈ and, the memory array regions R₁-R₄.

Each one of the memory regions R₁-R₄ is here comprised of SDRAM memorychips, as noted above. Each one of these regions R₁-R₄ is coupled to thefour crossbar switches 5004 ₁-5004 ₄ through a pair of memory regioncontroller, herein referred to as logic sections, here logic sections5010 ₁, 5010 ₂; . . . 5010 ₇, 5010 ₈, respectively. Each logic section5010 ₁-5010 ₈ is dual ported, (i.e., Port_A, (A) and Port_B, (B)) witheach port being coupled to one of the crossbar switches. The two logicsections 5010 ₁, 5010 ₂; . . . 5010 ₇, 5010 ₈(i.e., region controllers)associated with one of the memory regions R₁-R₄, respectively, sharecontrol of the SDRAM in such memory region. More particularly, and aswill be described in more detail below, each pair of logic section, suchas for example pair 5010 ₁ and 5010 ₂, share a common DATA port ofmemory array region R₁. However, each one of the logic sections 5010 ₁and 5010 ₂ is coupled to a different control port P_(A) and P_(B),respectively, of memory array region R₁, as indicated.

More particularly, each one of the crossbar switches 5004 ₁-5004 ₄ has,here, four lower ports 5008 ₁-5008 ₄ and four upper ports 5006 ₁-5006 ₄.Each one of the four upper ports 5006 ₁-5006 ₄, is, as noted above,coupled to a corresponding one of the four sets S₁-S₄, respectively, offour of the S/P converters. As noted above, the cache memory board 220 ₁also includes eight logic sections coupled 5010 ₁-5010 ₈ (to bedescribed in detail in connection with FIGS. 13A, 13B, 13C, 13D and 13E)as well as the four memory array regions R₁-R₄. An exemplary one of thememory array regions R₁-R₄ is described in connection with FIG. 6 ofU.S. Pat. No. 5,943,287. As described in such U. S. Patent, each one ofthe memory array regions includes a pair of redundant control portsP_(A), P_(B) and a data/chip select port (here designated as DATA). Asdescribed in such U. S. Patent, data may be written into, or read from,one of the memory array regions by control signals fed to either portP_(A) or to port P_(B). In either case, the data fed to, or read from,the memory array region is on the common DATA port.

An exemplary one of the logic sections 5010 ₁-5010 ₈ will be discussedbelow in detail in connection with FIGS. 13A-15E and an exemplary one ofthe crossbar switches 5004 ₁-5004 ₄ in the logic networks 221 _(1B)-221_(2A) will be discussed below in detail in connection with FIGS. 10-12D.Suffice it to say here, however, each one of the memory array regionsR₁-R₄ is coupled to a pair of the logic sections 5010 ₁, 5010 ₂; 5010 ₃,5010 ₄; 5010 ₅, 5010 ₆; 5010 ₇, 5010 ₈, respectively, as shown. Moreparticularly, each one of the logic sections 5010 ₁, 5010 ₂; 5010 ₃,5010 ₄ 5010 ₅, 5010 ₆; 5010 ₇, 5010 ₈ includes: a pair of upper ports,Port_A (A), Port_B (B); a control port, C; and a data port, D, asindicated. The control port C of one each one of the logic sections 5010₁, 5010 ₃, 5010 ₅, 5010 ₇, is coupled to port P_(A) of a correspondingone of the four memory array regions R₁-R₄. In like manner, the controlport C of one of each one of the logic sections 5010 ₂, 5010 ₄, 5010 ₆,5010 ₈ is coupled to port P_(B) of a corresponding one of the fourmemory array regions R₁-R₄, respectively as shown. Thus, each one of thememory array regions R₁-R₄ is coupled to a redundant pair of the logicsections 5010 ₁, 5010 ₂; 5010 ₃, 5010 ₄; 5010 ₅, 5010 ₆; 5010 ₇, 5010 ₈,respectively. The data ports D of logic section pairs 5010 ₁, 5010 ₂;5010 ₃, 5010 ₄; 5010 ₅, 5010 ₆; 5010 ₇, 5010 ₈, respectively, arecoupled together and to the DATA port of a corresponding one of thememory regions, R₁-R₄, respectively, as indicated.

It should be noted that each one of the crossbar switches 5004 ₁-5004 ₄is adapted to couple the upper ports 5006 ₁-5006 ₄ thereof to the lowerports 5008 ₁-5008 ₄ thereof selectively in accordance with a portion(i.e., a “tag” portion) of the information fed to the crossbar switch.In response to such “tag” portion, a transfer of information between aselected one of the memory array regions R₁-R₄ and a selected the of thedirectors coupled to the crossbar switch is enabled. The memory controlportion (e.g., read, write, row address select, column address select,etc.) of the information passes between either port A or port B of alogic sections 5010 ₁, 5010 ₃, 5010 ₅, 5010 ₇, and port P_(A) of thememory array region R₁-R₄ coupled to such logic section and the data(DATA) portion of the information passes to the DATA port of suchcoupled memory array region R₁-R₄, respectively. Likewise, the controlportion of the information passes between port A or port B of a logicsections 5010 ₂, 5010 ₄, 5010 ₆, 5010 ₈, and port P_(B) of the memoryarray region R₁-R₄ coupled to such logic section and the data portion ofthe information passes to the DATA port of such coupled memory arrayregion R₁-R₄, respectively.

Thus, each one of the logic sections 5010 ₁-5010 ₈ includes a pair ofredundant upper ports, A and B. The lower ports 5008 ₁-5008 ₄ ofcrossbar switch 5004 ₁, are coupled to the A port of logic sections 5010₁, 5010 ₃, 5010 ₅, and 5010 ₇, respectively, while the lower ports 5008₁-5008 ₄ of crossbar switch 5004 ₂ are coupled to the B port of logicsections 5010 ₁, 5010 ₃, 5010 ₅, and 5010 ₇, respectively. The lowerports 5008 ₁-5008 ₄ of crossbar switch 5004 ₃ are coupled to the A portof logic sections 5010 ₁, 5010 ₃, 5010 ₅, and 5010 ₇, respectively,while the lower ports 5008 ₁-5008 ₄ of crossbar switch 5004 ₄ arecoupled to the B port of logic sections 5010 ₂, 5010 ₄, 5010 ₆, and 5010₈, respectively.

As noted above in connection with FIG. 2, each one of the host computerprocessors 121 ₁-121 ₃₂ is coupled to here a pair (but not limited to apair) of the front-end directors 180 ₁-180 ₃₂, to provide redundancy inthe event of a failure in one of the front end-directors 181 ₁-181 ₃₂coupled thereto. Likewise, the bank of disk drives 140 has a pluralityof, here 32, disk drives 141 ₁-141 ₃₂, each disk drive 141 ₁-141 ₃₂ iscoupled to here a pair (but not limited to a pair) of the back-enddirectors 200 ₁-200 ₃₂, to provide redundancy in the event of a failurein one of the back-end directors 200 ₁-200 ₃₂ coupled thereto. Thus, thesystem has redundant front-end processor pairs 121 ₁, 121 ₂ through 121₃₁, 121 ₃₂ and redundant back-end processor pairs 141 ₁, 141 ₂ through141 ₃₁, 141 ₃₂. Considering the exemplary logic network 220 ₁ shown inFIGS. 9A, 9B and 9C, as noted above in connection with FIG. 8B,redundant front-end processor pairs 121 ₁ and 121 ₂, are able to becoupled to ports M_(A1) and M_(B1) of a cache memory board. Thus, theports M_(A1) and M_(B1) may be considered as redundant memory boardports. In like manner, the following may be considered as redundantmemory ports because the are able to be coupled to a pair of redundantprocessors: M_(A2) and M_(B2); M_(A3) and M_(B3); M_(A4) and M_(B4);M_(A5) and M_(B5); M_(A6) and M_(B6); M_(A7) and M_(B7); and, M_(A8) andM_(B8). It is noted that ports M_(A1) and M_(B1); M_(A2) and M_(B2);M_(A3) and M_(B3); M_(A4) and M_(B4) are coupled to the front-endprocessors through front-end directors and ports M_(A5) and M_(B5);M_(A6) and M_(B6); M_(A7) and M_(B7); M_(A8) and M_(B8) are coupled tothe disk drives through back-end directors.

Referring again to FIGS. 9A, 9B and 9C, from the above it should benoted then that logic networks 221 _(1B) and 221 _(1A) may be consideredas a pair of redundant logic networks (i.e., pair 1) because they areable to be coupled to redundant pairs of processors, here front-endprocessors. Likewise, logic networks 221 _(2B) and 221 _(2A) may beconsidered as a pair of redundant logic networks (i.e., pair 2) becausethey are able to be coupled to redundant pairs of disk drives. Further,logic network 221 _(1B) of pair 1 is coupled to upper port A of logicsections 5010 ₁, 5010 ₃, 5010 ₅, and 5010 ₇ while logic network 221_(1A) of pair 1 is coupled to port A of the logic sections 5010 ₂, 5010₄, 5010 ₆, and 5010 ₈. Logic network 221 _(2B) of pair 2 is coupled toport B of logic sections 5010 ₁, 5010 ₃, 5010 ₅, and 5010 ₇ while logicnetwork 221 _(2A) of pair 2 is coupled to port B of the logic sections5010 ₂, 5010 ₄, 5010 ₆, and 5010 ₈.

Thus, from the above it is noted that ports M_(B1)-M_(B4), which arecoupled to one of a pair of redundant processors, are adapted to becoupled to one of the ports in a pair of redundant control ports, hereport P_(A) of the four memory array regions R₁-R₄ while portsM_(A1)-M_(A4), of the other one of the pair of redundant processors areadapted to be coupled to the other one of the ports of the redundantcontrol ports, here port P_(B) of the four memory array regions R₁-R₄.Likewise, ports M_(B5)-M_(B8), which are coupled to one of a pair ofredundant processors, are adapted to be coupled to one of the ports in apair of redundant control ports, here port P_(A) of the four memoryarray regions R₁-R₄ while ports M_(A5)-M_(A8), of the other one of thepair of redundant processors are adapted to be coupled to the other oneof the ports of the redundant control ports, here port P_(B) of the fourmemory array regions R₁-R₄.

Thus, the memory board 220 ₁ (FIGS. 9A, 9B and 9C) is arranged with apair of independent fault domains: One fault domain, Fault Domain A, isassociated with logic networks 221 _(1B) and 221 _(2B), logic sections5010 ₁, 5010 ₃ 5010 ₅, 5010 ₇, and ports P_(A) of the memory arrayregions R₁-R₄ and, the other fault domain, Fault Domain B, is associatedwith logic networks 221 _(1A) and 221 _(2A), logic sections 5010 ₂, 5010₄, 5010 ₆, 5010 ₈ and port P_(B) of the memory array regions R₁-R₄. Thelogic in each one of the fault domains is operated by a correspondingone of a pair of independent clocks, Clock 1 and Clock 2 (FIGS. 9A, 9Band 9C). More generally, a fault domain is defined as a collection ofdevices which share one or more common points of failure. Here, FaultDomain A includes: logic networks 221 _(1B), 221 _(2B) (i.e., the S/Psand crossbar switches 5004 ₁-5004 ₂ therein) and logic sections 5010 ₁,5010 ₃, 5010 ₅, 5010 ₇, such devices being indicated by lines whichslope from lower left to upper right (i.e., ///). The other faultdomain, Fault Domain B, includes: logic networks 221 _(1A), 221 _(AB)(i.e., the S/Ps and crossbar switches 5004 ₃-5004 ₄ therein) and logicsections 5010 ₂, 5010 ₄, 5010 ₆, 5010 ₈, such devices being indicated bylines which slope from upper left to lower right (i.e., \\\\). It isnoted from FIGS. 9A, 9B and 9C that port P_(A) of each one of the memoryarray regions R₁-R₄ is coupled to Fault Domain A while port P_(B) iscoupled to fault domain B. Thus, each one of the fault domains includesthe devices used to couple one of a pair of redundant processors to oneof a pair of redundant control ports P_(A), P_(B) of the memory arrayregions R₁-R₄ and the other fault domain includes the devices used tocouple the other one of the pair of redundant processors to the otherone of a pair of redundant control ports P_(A), P_(B) of the memoryarray regions R₁-R₄. As noted above each fault domain operates with aclock (i.e., clock 1, clock 2) separate from and independent of theclock used to operate the other fault domain.

Referring now to FIG. 10, an exemplary one of the crossbar switches 5004₁-5004 ₄, here crossbar switch 5004 ₁ is shown in detail to include fourupper port interface sections A-D and lower port interface sections W-Z.The details of an exemplary one of the upper port interface sectionsA-D, here upper port interface section A, will be described in moredetail in connection with FIGS. 11A, 11B, 11C and 11D and the details ofan exemplary one of the lower port interface sections W-Z, here lowerport interface section W, will be described in more detail in connectionwith FIGS. 12A, 12B, 12C and 12D. The function of the exemplary crossbarswitch 5004 ₁ is to mediate the information cycle at the request of aninitiating one of the directors coupled to one of the upper 5006 ₁-5006₄ and one logic section 5010 ₁-5010 ₈ indicated by the “tag” portion ofthe information (FIG. 16).

More particularly, the crossbar switches request, negotiate, and theneffect a transfer between the upper thereof 5006 ₁-5006 ₄ and the lowerports 5008 ₁-5008 ₄ thereof in a manner to be described below. Sufficeit to say here, however, that the upper interface section A-D handle theprotocol between the director requesting a information cycle and thememory board 220 ₁ (FIG. 8). It also provides a control and datainterface to the serial-to-parallel (S-P) converters (e.g.,serializer-deserializer). These interface sections A-D are alsoresponsible for generating parity across the address, control, DATA, andCRC received from the director. There are here two parity bits, one percycle as described in co-pending patent application entitled “FaultTolerant Parity Generation” filed May 20, 1999, Ser. No. 99/315,437, andassigned to the same assignee as the present invention, the entiresubject matter being incorporated herein by reference. As described insuch patent application, the parity is generated such that one byte hasodd parity and the other byte has even parity. The sense of these paritybits alternate on successive clocks.

The lower port interface sections W-Z provides address, control, DATAand routing to one of the four of the logic sections 5010 ₁-5010 ₈(FIGS. 9A, 9B and 9C) in a manner to be described. Each one of the lowerinterface sections W-Z is adapted to couple a corresponding one of thefour memory array regions R₁-R₄ (FIGS. 9A, 9B and 9C), respectively, vialogic sections 5010 ₁-5010 ₈. Each one of the four lower interfacesections W-Z independently acts as an arbiter between the four upperinterface sections A-D and the logic section 5010 ₁-5010 ₈ coupledthereto. This allows for simultaneous transfers (i.e., informationcycles) to multiple memory array regions R₁-R₄ from multiple upperinterface sections A-D. The upper interface section A-D are singlethreaded, i.e., one information cycle must be complete before anotherinformation cycle is allowed to the same memory array regions R₁-R₄.

The lower interfaces W-Z deliver control, address and the “tag” field(to be described in more detail below) to the logic section 5010 ₁-5010₈. The parity across these fields are generated in the upper interfacesections A-D and then pass unmodified such that the memory array regioncan check for alternating parity sense. For write transfers, the lowerinterface sections W-Z also deliver the write data to the memory arrayregion, checking for correct CRC across the data. If any error isdetected, and if the control field indicates a “Wait-and-Validate”process to be described, the parity of the last double byte of data iscorrupted (e.g., a fault is induced in the parity (p) thereof) such thatthe logic section 5010 ₁-5010 ₈ coupled thereto detects the corruptedparity and inhibits execution of the information cycle. Otherwise, thealternating parity of the data is unmodified. For read transfers, thelower interface sections W-Z accept the data from the memory arrayregions R₁-R₄ via the logic sections 5010 ₁-5010 ₈, check thealternating parity, and generates CRC to be returned to the director.

More particularly, assume for example that information at upper port5006 ₄ (FIGS. 9A, 9B and 9C) of crossbar switch 5004 ₄ is to betransferred to memory array region R₁. Referring to FIG. 10 anegotiation, i.e., arbitration, must be made by lower port interface Was a result of a request made by the upper port interface section D ofcrossbar switch 5004 ₄ to section interface W thereof. When interfacesection W is available to satisfy such request, (i.e., not satisfyingrequest from other one of the upper port interface sections A-C)interface W issues a grant to upper interface section D.

Thus, each one of the upper port sections A-D sends requests signals(REQs) to the lower port sections W-Z when such upper port sections A-Dwants access to (i.e. wants to be coupled to) such lower port sections.Conversely, each one of the upper port sections A-D receives grantsignals (GR) from the lower port sections W-Z when such lower portsections W-Z grants access to (i.e., wants to be coupled to) such upperport sections A-D. The request (REQ) and grant (GR) signals, produced byand received from tile upper port sections A-D and lower port sectionsW-Z are as follows:

UP R R G G G G PER PORT EQ EQ EQ EQ R R R R SECTION R RZA G G GYA G A WAXA YA WA XA ZA UP R R G G G G PER PORT EQ EQ EQ EQ R R R R SECTION R RZBG G GYB G B WB XB YB WB XB ZB UP R R G G G G PER PORT EQ EQ EQ EQ R R RR SECTION R RZC G G GYC G C WC XC YC WC XC ZC UP R R G G G G PER PORT EQEQ EQ EQ R R R R SECTION R RZD G G GYD G D WD XD YD WD XD ZD LO R R G GG G WER EQ EQ EQ EQ R R R R PORT R RWD G G GWC G SECTION WA WB WC WA WBWD W LO R R G G G G WER EQ EQ EQ EQ R R R R PORT R RXD G G GXC G SECTIONXA XB XC XA XB XD X LO R R G G G G WER EQ EQ EQ EQ R R R R PORT R RYD GG GYC G SECTION YA YB YC YA YB YD Y LO R R G G G G WER EQ EQ EQ EQ R R RR PORT R RZD G G GZC G SECTION ZA ZB XC ZA ZB ZD Z

where:

For upper port section A:

RWA is a request signal sent by upper port section A to lower portsection W;

RXA is a request signal sent by upper port section A to lower portsection X;

RYA is a request signal sent by upper port section A to lower portsection Y:

RZA is a request signal sent by upper port section A to lower portsection Z;

GWA is a grant signal from lower port section W to upper port section A;

GXA is a grant signal from lower port section X to upper port section A;

GYA is a grant signal from lower port section Y to upper port section A;

GZA is a grant signal from lower port section Z to upper port section A;

For upper port B:

RWB is a request signal sent by upper port section B to lower portsection W;

RXB is a request signal sent by upper port section B to lower portsection X;

RYB is a request signal sent by upper port section B to upper portsection Y:

RZB is a request signal sent by upper port section B to lower portsection Z;

GWB is a grant signal from lower port section W to upper port section B;

GXB is a grant signal from lower port section X to upper port section B;

GYB is a grant signal from lower port section Y to upper port section B;

GZB is a grant signal from lower port section Z to upper port section B;

and so forth for the remaining upper and lower port sections C-D andW-Z.

Each one of the upper port sections A-D has four ports A₁-A₄, throughD₁-D₄, respectively, as shown. Each one of the lower port sections W-Zhas four ports W₁-W₄, through Z₁-Z₄, respectively, as shown. Ports A₁-A₄are connected to ports W₁-Z₁, respectively, as shown. In like manner,Ports B₁-B₄ are connected to ports W₂-Z₂, respectively, as shown, portsC₁-C₄ are connected to ports W₃-Z₃, as shown, and Ports D₁-D₄ areconnected to ports W₄-Z₄, as shown. Lower ports 5008 ₁-5008 ₄ areconnected to lower port sections W-Z, respectively, as shown.

As noted above, an exemplary one of the upper port interface sectionsA-D and an exemplary one of the lower port interface sections W-Z willbe described in more detail in connection with FIGS. 11A-11D and12A-12D, respectively. Suffice it to say here, however, that informationfed to port 5006 ₁ is coupled to ports 5008 ₁-5008 ₄ selectively inaccordance with a “tag” portion such information. In a reciprocalmanner, information fed to port 5008 ₁ is coupled to ports 5006 ₁-5006 ₄selectively in accordance with the “tag” portion in such information.Further, ports 5006 ₂-5006 ₄ operate in like manner to port 5006 ₁, sothat information at such ports 5006 ₂-5006 ₄ may be coupled to ports5008 ₁-5008 ₄. Still further, ports 5008 ₂-5008 ₄ operate in like mannerto port 5008 ₁, so that information at such ports 5008 ₂-5008 ₄ may becoupled to ports 5006 ₁-5006 ₄. It should also be noted that informationmay appear simultaneously at ports 5008 ₁-5008 ₄ with the information atone of such ports being coupled simultaneously to one of the ports 5006₁-5006 ₄ while information at another one of the ports 5008 ₁-5008 ₄ iscoupled to a different one of the ports 5006 ₁-5006 ₄. It is also notedthat, in a reciprocal manner, information may appear simultaneously atports 5006 ₁-5006 ₄ with the information at one of such ports beingcoupled simultaneously to one of the ports 5008 ₁-5008 ₄ and withinformation at another one of the ports 5006 ₁-5006 ₄ being coupled to adifferent one of the ports 5008 ₁-5008 ₄.

Referring now to FIGS. 11A, 11B, 11C and 11D, an exemplary one of theupper port interface sections A-D, here upper port interface section Ais shown in more detail. It is first noted that the information at port5006 ₁ includes: the “tag” portion referred to above; an address CRCADDR_CRC portion, an address ADDR portion, a memory control portion(i.e., read/write, transfer length, “Wait and Validate”, etc.); a dataportion, (DATA); and a DATA Cyclic Redundancy Check (CRC) portion(DATA_CRC).

The “tag” portion includes: a two bit word indicating the one of thefour memory array regions R₁-R₄ where the data is to be stored/read; athree bit word indicating the one of the eight memory boards having thedesired array region R₁-R₄; a four bit word indicating the one of the 16director boards 190 ₁-190 ₈, 210 ₁-210 ₈ (FIG. 8) having the directorwhich initiated the transfer; a two bit word indicating which one of.thefour directors on such one of the director boards is making therequested data transfer; and a five bit random number designating,(i.e., uniquely identifying) the particular information cycle.

The information described above passing from the director to thecrossbar switch (i.e., the “tag”, the ADDR_CRC, the ADDR, the memorycontrol, the DATA, and the DATA_CRC) for the entire information cycle(FIG. 17) are successively stored in a register 5100, in response toclock pulses Clock I, in the order described above in connection withFIG. 17. Tile information stored in the register 5100 is passed to aparity generator (PG) 5102 for appending to such information a byteparity (p). After passing through the parity generator (PG) 5102, thedifferent portions of the information are stored in registers 5104₁-5104 ₆, as follows: Register 5104 ₁ stores the DATA_CRC portion (withthe generated parity); register 5104 ₂, here a FIFO, stores the dataportion, DATA, (with the generated parity); register 5104 ₃ stores thememory control portion (with the generated parity); register 5104 ₄stores the address ADDR portion (with the generated parity), register5104 ₅ stores the address ADDR_CRC portion (with the generated parity);and register 5104 ₆ stores the “tag” portion (with the generated parity)in the order shown in FIG. 17. Each clock pulse (Clock 1 or Clock 2)results in one of the words described above in connection with FIG. 17.Here, each word has two bytes and is stored in register 5100. The wordstored in register 5100 is then shifted out of register 5100 with thenext clock pulse, as new information becomes stored in such register5100.

The portions stored in the registers 5104 ₁-5104 ₄ and 5104 ₆ (notregister 5104 ₅ which stores ADDR_CRC) are fed to selectors 5106 ₁-5106₄, and 5106 ₆, respectively, as indicated. An exemplary one of theselectors 5106 ₁-5106 ₄, and 5106 ₆, here selector 5106 ₆ is shown toinclude four registers 5108 ₁-5108 ₄. The four registers 5108 ₁-5108 ₄are connected to the same input port I of the selector 5106 ₆ to therebystore four copies of the information portion, here the “tag” portion,fed to such input port I in this example. The output of each of the fourregisters 5108 ₁-5108 ₄ is fed to a corresponding one of four gatedbuffers 5110 ₁-5110 ₄, respectively, as indicated. With such anarrangement, one of the stored four copies is coupled to a selected oneof the output ports A₁-A₄ selectively (and hence to ports W₁-Z₁,respectively) in accordance with enable memory control signals on linesEAW-EAZ as a result of decoding the two-bit portion of “tag” indicatingthe selected one of the four memory, array regions R₁-R₄. Moreparticularly, each one of the lines EAW-EAZ is coupled to acorresponding one of the enable inputs of the four gated buffers 5110₁-5110 ₄, respectively, as indicated.

More particularly, as noted above, the “tag” includes 2 bits whichindicates the one of the four memory array regions R₁-R₄ which is toreceive the information at port 5006 ₁ (i.e., the “tag”, the ADDR_CRC,the ADDR, the memory control, the DATA, and the DATA_CRC). The “tag” isfed to a memory control logic/ADDR_CRC checker 5112. In response to thistwo bit portion of the “tag”, the memory control logic/ADDR CRC checker5112 activates one of the four lines EAW-EAZ to thereby enable aselected one of the four copies stored in the four registers 5108 ₁-5108₄ to pass to one of the ports A₁-A₄. It is noted that the lines EAW-EAZare also fed to selectors 5106 ₁-5106 ₅ in a similar manner with theresult that the information at port 5006 ₁ (i.e., the “tag”, theADDR_CRC, the ADDR, the memory control, the DATA, and the DATA_CRC)portions Data CRC, Data, memory control, ADDR, and ADDR_CRC is fed tothe same selected one of the ports A₁-A₄ and thus to the one of the fourmemory array regions R₁-R₄ described by the two-bit portion of the“tag”.

It is noted that the upper port section A also includes a memory boardchecker 5114. Each of the here eight memory board 220 ₁-220 ₈ (FIG. 8)plugs into the backplane 302 as discussed above in connection with FIG.3. As noted above, here the backplane 302 is adapted to a plurality of,here up to eight memory boards. Thus, here the backplane 302 has eightmemory board slots. Pins P₁-P₃ (FIGS. 9A, 9B and 9C) are provided foreach backplane 320 memory board slot and produce logic voltage levelsindicating the slot position in the backplane. Thus, here the slotposition may be indicated with the logic signals on the three pins P₁-P₃to produce a three bit logic signal representative of the backplane slotposition. Referring again to FIGS. 9A, 9B and 9C, the exemplary memoryboard 220 ₁ is shown plugged into a slot in the backplane 302. As notedabove, the slot has pins P₁-P₃ which provides the slot position threebit logic signal indicative of the slot or “memory board” number in thebackplane. The logic signals produced by the pins P₁-P₃ are fed to thememory board checker 5114 (FIGS. 11A, 11B, 11C and 11D). Also fed to thememory board checker 5114 are the 3-bits of the “tag” which indicatesthe one of the memory array boards which is to receive the data (i.e., a3-bit “memory board code”). If the three bit memory board indicationprovided by “tag” is the same as the backplane slot or “memory boardnumber” indication provided by the pins P₁-P₃, the director routed theinformation cycle to the proper one of the eight memory boards and such“accept” indication is provided to the decode logic/ADDR CRC checker5112 via line A/R. On the other hand, if the three bit memory boardindication provided by “tag” is different from the backplane slotindication provided by the pins P₁-P₃, the information cycle was notreceived by the correct one of the memory boards and such “reject”indication is provided to the decode logic/ADDR CRC checker 5112 vialine A/R. When a reject indication is provided to the decode logic/ADDRCRC checker 5112, the intended transfer in prevented and the indicationis provided by the decode logic/ADDR CRC checker 5112 to the initiatingdirector via the A/R line. Thus, if the “memory board number” providedby pins P₁-P₃ does not match the “memory board code” contained in the“tag” the transfer request from the director is rejected and such errorindication is sent back to the director. In this manner, a routing errorin the director is detected immediately and is not propagated along.

On the other hand, if the “memory board number” and the “memory boardcode” do match, the crossbar switch will forward the requested transferto one of the four memory regions (i.e., the “memory region number”,R₁-R₄) designated by the “tag”.

The decode logic and ADDR_CRC checker 5112 also produces load signalsL₁-L₆ to the registers 5104 ₁-5104 ₆, respectively, in response to the“start-frame” signal in word 0 described above in connection with FIG.16.

Also fed to the decode logic/ADDR_CRC checker 5112 is the ADDR_CRCportion stored in registers 5104 ₃ 5104 ₆ (i.e., control, ADDR,ADDR_CRC, and “tag”). The decode logic/ADDR_CRC 5112 performs a check ofthe CRC of the control, ADDR, ADDR_CRC, and “tag” and if such checker5112 detects an error such error is reported back to the transferinitiating director via line ADDR_CRC_CHECK, as indicated. Detection ofsuch an ADDR_CRC_CHECK error also results in termination of thetransfer.

When data is read from a selected one of the memory array region R₁-R₄as indicated by the “tag” stored in register 5104 ₆, the decodelogic/ADDR_CRC checker 5112 activates the proper one of the linesEAW-WAZ to coupled the proper one of the ports A₁-A₄ coupled to suchselected one of the memory array regions R₁-R₄ to a register 5120. Thus,read data passes via selector 5118 to the register 5120 and is then sentto the transfer-requesting director via pot 5006 ₁.

It is noted that the decode logic and ADDR CRC checker 5112 in upperport interface logic A also produces request signals RWA, RXA, RYA, andRZA and sends such request signal to lower port sections W-Z,respectively. Such requests are fed to an arbitration logic 5114 (FIGS.12A, 12B, 12C and 12D) included within each of the lower port sectionsW, X, Y and Z, respectively. Thus, because the other upper port sectionsB-D operate in like manner to upper port section A, the arbitration 5114in lower port interface section W may receive requests RWB, RWC, and RWDfrom such other upper port sections B-D, respectively. In accordancewith a predetermined arbitration rule, such as, for example, first-come,first-served, the arbitration logic 5114 of lower port interface sectionW grants for access to lower port 5008 ₁ of lower port section W to oneof the requesting upper port sections A-D via a grant signal on one ofthe lines GWA, GWB, GWC and GWD, respectively.

Thus, referring again to FIGS. 11A, 11B, 11C and 11D, the decodelogic/CRC ADR checker 5112 issues a request on line RWA when port 5008 ₁(FIG. 10) desires, based on the two bit information in the “tag”, memoryarray region R₁ (FIGS. 9A, 9B and 9C). In like manner, if memory arrayregions R₂-R₄ are indicted by the “tag”, requests are made by the upperport section on lines RXA, RYA, RZA, respectively. The other upper portsections B-D operate in like manner. The grants (GR) produced by thelower port sections W, X, Y and Z are fed to the upper port sections A-Das indicated above. Thus, considering exemplary upper port section A(FIGS. 11A, 11B, 11C and 11D), the grant signals from lower portsections W-Z are fed to the decode logic/CRC checker 5112 therein onlines GWA, GXA, GYA and GZA, respectively. When a grant on one of thesefour lines GWA, GXA, GYA and GZA is received by the decode logic/CRCchecker 5112, such checker 5112 enables the gating signal to be producedon the one of the enable lines EAW, EAX, EAY, EAZ indicated by the “tag”portion. For example, if the “tag” indicates that memory array region R₃(which is adapted for coupling to port 5008 ₃ of lower port section Y)the checker 5112 issues a request on line RYA. When after thearbitration logic 5114 in section Y determines that lower port logic Ais to be granted access to port 5008 ₃, such lower port section Y issuesa grant signal on line GYA. In response to such grant, the checker 5112issues an enable signal on line EAY to thereby enable information topass to port A₃ (FIGS. 11A, 11B, 11C and 11D).

In a reciprocal manner, when data is to be transferred from a memoryarray region to the requesting director, the information sent by therequesting director is processed as described above. Now, however, thechecker 5112 sends a control signal to one of the lines EAW-EAZ toselector section 5118 to enable data on one of the ports A₁-A₄ coupledto the addressed memory array regions R₁-R₄ to pass to register 5120 andthen to upper port 5006 ₁.

Referring now to FIGS. 12A, 12B, 12C and 12D, exemplary lower portsection W is shown to include arbitration logic 5114 described above,and the selector 5120 fed by signals on ports W₁-W₄. (Referring again toFIG. 10, ports W₁-W₄ are coupled to ports A₁, B₁, C₁ and D₁,respectively, of upper port interface sections A-D, respectively.) Thus,when the arbitration logic 5114 grants access to one of the upper portsections A-D, the decoder 5122 decodes the grant information produced bythe arbitration logic and produces a two bit control signal for theselector 5120. In response to the two bit control signal produced by thedecoder 5122, the selector couples one of the ports W₁-W₄ (and hence oneof the upper port sections A-D, respectively), to the output of theselector 5120 and hence to lower port 5008 ₁ in a manner to bedescribed.

As noted above, the communication between any port M_(B1)-M_(B8),M_(A1)-M_(A8) and its corresponding crossbar switches 5004 ₁-5004 ₄ isprotected by Cyclic Redundancy Check (CRC) defined by CCITT-V.41. Thecommunication between a crossbar switch 5004 ₁-5004 ₄ and itscorresponding memory array region R₁-R₄ is protected by byte parity (p).There is a pipelined architecture from the port M_(B1)-M_(B8),M_(A1)-M_(A8), and through the crossbar switch, and through the logicsections 5010 ₁-5010 ₈.

The nature of CRC calculation is such that an error in the data is notdetected until the entire transfer is completed and the checksum of theCRC is known. In the case of a write of data into the memory, by thetime the CRC is checked, most of the data is already through thepipeline and written into memory.

Here, the memory control field has a specific bit “Wait and Validate” inthe control word 1 in FIG. 16 which is at the director's control. If thebit is set, the logic sections 5010 ₁-5010 ₈ buffers the entireinformation cycle, pending the CRC calculation, performed at the lowerport interface sections W-Z. If the CRC check indicates no CRC error,then the data is written into the memory array region. If the CRC checkdoes indicate an error, then the memory array region is informed of theerror, here by the lower interface section W-Z corrupting the data intoa fault. Such fault is detected in the logic section 5010 ₁-5010 ₈ andsuch information is prevented from being stored in the memory regionR₁-R₄, in a manner to be described. Suffice it to say here, however,that this “Wait and Validate” technique enables the director to flagcertain data transfers as critical, and if an error occurs, preventscorruption of the data stored in the memory array. That is, the datahaving a CRC error is detected and prevented from being stored in thememory array region. For those transfers not indicated as critical bythe director, the “Wait and Validate” bit is not set thereby maximumperformance of the memory is obtained.

More particularly, the DATA, memory control, ADDR, and “tag” portions(with their byte parity (p) generated by parity generator 5102 (FIGS.11A, 11B, 11C and 11D)) of the information coupled to the output ofselector 5120 is stored in the register 5124. As noted above inconnection with FIG. 16, the DATA_CRC portion (i.e., the words X and Y)occurs after the last DATA word.

Thus, as the words in the DATA clock through register 5124 they passinto the DATA_CRC checker 5132 where the CRC of the DATA is determined(i.e., the DATA_CRC checker 5132 determine X and Y words of the DATA fedto such checker 5132). The actual X and Y words (i.e., DATA_CRC storedin register 5128, both content (n) and parity (p)) are storedsuccessively in register 5128 and are then passed to checker 5132 wherethey are checked against the X and Y words determined by the checker5132. As noted above, the DATA has appended to it its parity (p). Thus,the “information” whether in register 5124 or register 5128 has acontent portion indicated by “n” and its parity indicated by “p”. Thus,the DATA_CRC register 5128 includes the DATA_CRC previously stored inregister 5104 ₁ (FIGS. 11A, 11B, 11C and 11D) (i.e., the content portiondesignated by “n”) and its parity (designated by “p”). The DATA, memorycontrol, ADDR, and “tag” portions, (with their parity (p) (i.e., content“n” plus its appended parity “p”) stored in register 5124 may be coupledthrough a selector 5149 through one of two paths: One path is a directpath when the “Wait and Validate” command is not issued by the director;and, a second path which includes a delay network 5130, here a threeclock pulse delay network 5130.

More particularly, it is noted that the DATA, control, ADDR, “tag”, bothcontent (n) and parity (p) are also fed to a DATA_CRC checker 5132. Alsofed to the DATA_CRC checker 5132 is the output of DATA_CRC register5128. The CRC checker 5132 checks whether the DATA_CRC (content “n” plusits parity “p”) is the same as the CRC of the DATA, such DATA havingbeen previously stored in register 5104 ₂ (FIGS. 11A, 11B, 11C and 11D),i.e., the content “n” plus its parity “p” of the DATA previously storedin register 5104 ₂ (FIGS. 11A, 11B, 11C and 11D). If they are the same,(i.e., no DATA_CRC_ERROR), a logic 0 is produced by the CRC checker5132. If, on the other hand, they are not the same, (i.e., aDATA_CRC_ERROR), the CRC checker 5132 produces a logic 1. The output ofthe Data_CRC checker 5132 thereby indicates whether there is an error inthe CRC of the DATA. Note that a DATA_CRC_ERROR is not known until threeclock cycles after the last sixteen-bit portion of the DATA (i.e., theword of the DATA, FIG. 16) is calculated due to the nature of the CRCalgorithm. Such indication is fed to a selector 5152 via an OR gate5141. If there is a DATA_CRC_ERROR, the “information” at the output ofthe delay network 5130 (i.e., the last word of the DATA (FIG. 16)) withits parity (p)) is corrupted. Here, the content (n) of such“information” (i.e., the “information” at the output of the delaynetwork 5130 (i.e., the last word of the DATA (FIG. 16))) is fed to asecond input I₂ of the selector 5140. The parity (p) of such“information” (i.e., the last word of the DATA (FIG. 16)) is fednon-inverted to one input of selector 5152 and inverted, via inverter5150, to a second input of the selector 5152. If there is aDATA_CRC_ERROR detected by data CRC checker 5132, the inverted parity ispassed through the selector 5152 and appended to the content portion (n)of the “information” (i.e., the last word of the DATA (FIG. 16))provided at the output of the delay network 5130 and both “n” andappended “p” are fed to the second input I₂ of selector 5140 therebycorrupting such “information”. It should be noted that the remainingportions of the information cycle (i.e., the memory control, address(ADDR), “tag”, and all but the last word of the DATA (FIG. 16)) passthrough the delay network 5130 without having their parity (p)corrupted.

If there is a no “Wait and Validate” transfer, logic decoder 5122selects the first input I₁ as the output of the selector 5140. If thereis a “Wait and Validate” transfer, the logic decoder 5122 selects thesecond input I₂ as the output of the selector 5140. It is noted,however, that that because the last word of DATA (FIG. 16) is delayedthree clock pulses (from Clock 1) by registers 5142, 5144, and 5146(such registers 5142, 5144 and 5146 being fed by such Clock 1), theDATA_CRC check is performed before the last word of the DATA appears atthe output of register 5146. Thus, the last word of the DATA iscorrupted in byte parity before being passed to the logic section 5010₁-5010 ₈. That is, because of the delay network 5130, the DATA_CRC isevaluated before the last word of the DATA has passed to port 5008 ₁.This corruption in parity (p), as a result of a detected DATA_CRC error,is detected by a parity checker 6106 (FIGS. 14A, 14B, 14C and 14D) inthe following logic section 5010 ₁-5010 ₈ in a manner to be described.Suffice it to say here, however, that detection of the parity error(produced by the detected CRC error) prevents such corrupted informationfrom storage in the SDRAMs.

On the other hand, if there is no DATA_CRC_ERROR (and no error in theparity of the DATA_CRC detected by the parity checker 6106 (FIGS. 14A,14B, 14C and 14D) in a manner to be described) the non-inverted parity(p) is appended to the “information” (i.e., DATA, memory control, ADDR,and “tag”) provided at the output of the delay network 5130 and suchinformation is fed to the proper memory address region R₁-R₄ asindicated by “tag”.

More particularly, it is noted that the selector 5140 is also fed the“information” (i.e., DATA, memory control, ADDR, and “tag”) without such“information” passing through the delay 5130. The director issuing thetransfer may not require that the transfer have the DATA_CRC checkresult preclude the writing to information into the memory (i.e., no“Wait and Validate”), in which case the “information” is passed directlythrough the selector 5140. On the other hand, if such DATA_CRC check isto be effected, the delay network 5130 output, with a possiblecorruption as described above, is passed through the selector 5140. Thedirector provides the indication as part of the control field in thedescribed “Wait and Validate” bit. Such bit is decoded by the logicdecoder 5122. In response to such director indication, a “Wait andValidate” control signal is sent by the logic decoder 5122 to theselector 5140.

As noted above, the communication between any port and its correspondingcrossbar switch is protected by Cyclic Redundancy Check (CRC) defined byCCITT-V.41. The communication between a crossbar switch and a memoryarray region R₁-R₄ is protected by byte parity (p). This implies thatthe crossbar switch must translate between CRC protection and parityprotection.

As a further check of the validity of the DATA CRC, the generated parityp of the CRC of Such DATA is checked. However, because the CRC isgenerated by the director, and the CRC parity is also generated by upperinterface section A-D, a CRC generation fault would yield anundetectable CRC parity fault.

It has been discovered that the parity (p) of the DATA_CRC must be thesame as the parity of the DATA parity (p). Thus, one merely has to checkwhether the parity of the DATA_CRC is the same as the parity of the DATAparity (p). Therefore, such detection DATA_CRC parity checking method isaccomplished without using the DATA_CRC itself.

More particularly, since the DATA over which the DATA_CRC is beingcalculated is already parity protected, one can use the DATA parity (p)to calculate the DATA_CRC parity: i.e., the DATA_CRC parity is equal tothe parity of all the DATA parity bits. Still more particularly, ifthere are N bytes of DATA:

[D(0), D(1), . . . D(N−1)]

and each byte is protected by a parity bit p, then the DATA_CRC parityis the parity of

[p(0), p(1), . . . p(N−1)].

Thus, if there is a fault in the generation of the DATA_CRC, it isimmediately detected and isolated from the director.

Thus, the exemplary lower port interface section W (FIGS. 12A, 12B, 12Cand 12D) includes a parity generator made up of an exclusive OR gate5134 and register 5136 arranged as shown fed by the parity (p) of theDATA portion stored in register 5124. The generated parity p is fed to acomparator 5138 along with the parity (p) of the DATA_CRC (i.e.,DATA_CRC_PARITY), as indicated. If the two are the same at the end ofthe DATA portion of the information cycle (FIG. 16), a logic 0 isproduced by the comparator 5138 and such logic 0 passes to the selector5152 to enable the non-inverted parity to pass through such selector5152. If there is an error in the parity bit of the CRC, a logic 1 isproduced by the comparator 5138 and the inverted parity is passedthrough the selector 5152. The logic 1 output of comparator 5138 passesthrough OR gate 5141 to couple the inverted parity (p) through selector5152 to append to the content port (n) of DATA control, ADDR, and “tag”at port I₂ of selector 5140. Thus, if there is either a DATA_CRC_ERRORor if DATA_CRC_PARITY is different from parity of the DATA_PARITY at theend of the DATA portion of the information cycle as indicated by asignal produced on line COMP_ENABLE by the logic decoder 5122, a logic 1is produced at the output of OR gate 5141 thereby coupling the invertedparity through selector 5152. Otherwise, the non-inverted parity passesthrough selector 5152. That is, the COMP_EN is produced at the end ofthe DATA in the information cycle (FIG. 16).

It is noted that information read from the memory region passes to aregister 5170 and a CRC generator 5172. The generated CRC is appended tothe information clocked out of the register 5170. Four copies of theinformation with appended CRC are stored in registers 5174 ₁-5174 ₄,respectively. In response to the “tag” portion fed to logic decoder5122, a selected one of the registers 5174 ₁-5174 ₄ is coupled to one ofthe port W₁-W₄ by selector 5180 and gates 5182 ₁-5182 ₄ in a mannersimilar to that described in connection with FIGS. 11A, 11B, 11C and11D.

Referring now to FIGS. 13A, 13B, 13C, 13D and 13E, a pair of the logicsections 5010 ₁-5010 ₈ (memory array region controllers), here logicsections 5010 ₁ and 5010 ₂ are shown. As noted above in connection withFIGS. 9A, 9B and 9C, both logic sections 5010 ₁ and 5010 ₂ are coupledto the same memory array region, here memory array region R₁. As wasalso noted above in connection with FIGS. 9A, 9B and 9C, the logicsection 5010 ₁ is in one fault domain, here fault domain A, and logicsection 5010 ₂ is in a different fault domain, here fault domain B.Thus, logic section 5010 ₁ operates in response to clock pulses fromClock 1 and logic section 5010 ₂ operates in response to clock pulsesfrom Clock 2.

As noted above, each logic section 5010 ₁-5010 ₈ (FIGS. 9A, 9B and 9C)includes a pair of upper ports, A and B, a control port C and a dataport D. Referring to FIGS. 13A, 13B, 13C, 13D and 13E, an exemplarylogic section 5010 ₁ is shown in detail to include a upper port Acontroller 6002A coupled to upper port A, a upper port B controller6002B coupled to upper port B, and a memory refresh section 6002R.

Both port A and port B controllers 5010 ₁, 5010 ₂ have access to thedata stored in the same memory array region R₁. Further, while each canprovide different, independent control and address information, (i.e.,memory control, ADDR, and “tag” (hereinafter sometimes referred to asADDR/CONTROL)), both share the same DATA port. As noted above, thedetails of the memory array region I are described in detail inconnection with FIG. 6 of U.S. Pat. No. 5,943,287. Thus, arbitration isrequired for access to the common memory array region R₁ when both theport A and port B controllers 5010 ₁ and 5010 ₂ desire access to thememory array region R₁. Further, the SDRAMs in the memory array regionR₁ require periodic refresh signals from the memory refresh section 6002R. Thus, access or request for, the memory array region R₁ may comefrom: the upper port A controller 6002A (i.e., REQUEST A); the upperport B controller 6002B (i.e., REQUEST B); and from the memory refreshsection 6002R (i.e., REFRESH REQUEST). These request arc fed to anarbitration logic 6004 included within the logic section 5010 ₁-5010 ₈.The arbitration sections 6004 ₁, 6004 ₂ in the redundant paired logicsections, here logic sections 5010 ₁, 5010 ₂, respectively, arbitrate inaccordance with an arbitration algorithm to be described and thereby toissue a grant for access to the memory array region R₁ to either: theupper port A controller 6002A (i.e., GRANT A); the upper port Bcontroller 6002B (i.e., GRANT B); or the memory refresh section 6002R(i.e., REFRESH GRANT).

Here, the arbitration algorithm is an asymmetric round robin sharing ofthe common memory array region R₁. The arbitration logic 6004 ₁, 6004 ₂and the algorithm executed therein will be described in more detail inconnection with FIGS. 15A, 15B, 15C, 15D and 15E. Suffice it to say herehowever that the arbitration grants access to the common memory arrayregion based on the following conditions:

Condition I—If both the logic sections 5010 ₁ and 5010 ₂ are operatingproperly (i.e., produce Memory Output Enable (MOE) and Memory RefreshEnable (MRE) signals, to be described, properly), the port A controller6002A memory refresh controller 6002R is used exclusively for memoryrefresh during the round-robin arbitration). Thus, there is asymmetricround robin arbitration because the memory refresh section 6002R oflogic section 5010 ₂ is not used when operating in this normal ConditionI. The states of the arbitration sequences are as follows:

State 1—The upper port A controller 6002A of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 2—The memory refresh section 6002R of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 3—The upper port B controller 6002B of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 4—The memory refresh section 6002R of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 4—A check is made as to whether the of logic section 5010 ₂requests access to the memory array region R₁. If such a request exist:

(a) The upper port A controller 6002A of logic section 5010 ₂ is grantedaccess to the memory array region R₁ if such access is requested;

(b) The upper port B controller 6002B of logic section 5010 ₂ is grantedaccess to the memory array region R₁ if such access is requested;

State 5—The process returns to State 1.

(It should be noted that the process uses the memory refresh section6002R of logic section 5010 ₁ but does not use the memory refreshsection 6002R of logic section 5010 ₂. Thus the round robin isasymmetric.)

Condition II—If the logic section 5010 ₂ is disabled (i.e., does notproduce MOE and MRE signals properly), the logic section 5010 ₂ is notpart of the round-robin arbitration and memory refresh is provided, asin Condition I, exclusively by the logic section 5010 ₁ memory refreshcontroller 6002R. The logic section 5010 ₁ no longer receives requestsignals FROM the logic section 5010 ₂. Also the logic section 5010 ₁ isgranted access to the memory array region R₁ all the time. Thus, thestates of the arbitration sequence are in Condition II as follows:

State 1—The upper port A controller 6002A of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 2—The memory refresh section 6002R of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 3—The upper port B controller 6002B of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 4—The memory refresh section 6002R of logic section 5010 ₁ isgranted access to the memory array region R₁;

State 5—The process returns to State 1.

Condition III—The logic section 5010 ₁ is disabled (i.e., does notproduce MOE and MRE signals properly) and thus the logic section 5010 ₁is not part of the round-robin arbitration. Memory refresh is providedexclusively by the memory refresh section 6002R (not shown) in the logicsection 5010 ₂. The logic section 5010 ₂ is granted access to the memoryarray region R₁ all the time. Thus the states of the arbitrationsequence in Condition III are as follows:

State 1—The upper port A controller 6002A of logic section 5010 ₂ isgranted access to the memory array region R₁;

State 2—The memory refresh section 6002R of logic section 5010 ₂ isgranted access to the memory array region R₁;

State 3—The upper port B controller 6002B of logic section 5010 ₂ isgranted access to the memory array region R₁;

State 4—The memory refresh section 6002R of logic section 5010 ₂ isgranted access to the memory array region R₁;

State 5—The process returns to State 1.

Condition IV-Reset (the arbitration is reset into Condition I fromeither Condition II or from condition III).

Referring again to FIGS. 13A, 13B, 13C, 13D and 13E, the arbitrationlogic 6004 ₁, 6004 ₂ in each one of the logic sections 5010 ₁, 5010 ₂produces: a memory output enable (MOE) signal; a memory refresh enable(MRE) signal (to be described in more detail in connection with FIGS.15A-15E and 19); and, a memory grant (MG) signal, (to be described inmore detail in connection with FIGS. 15A-15E and 19). Thus, logicsection 5010 ₁ produces a memory output enable signal MOEA (to bedescribed in more detail in connection with FIGS. 15A-15E and 19), amemory refresh enable signal MREA (to be described in more detail inconnection with FIGS. 15A-15E and 19) and a memory grant signal MGA (tobe described in more detail in connection with FIGS. 15A-15E and 19).Likewise, logic section 5010 ₂ produces a memory output enable signalMOEB (to be described in more detail in connection with FIGS. 15A-15Eand 19), a memory refresh enable signal MREB (to be described in moredetail in connection with FIGS. 15A-15E and 19) and a memory grantsignal MGB (to be described in more detail in connection with FIGS.15A-15E and 19). Suffice it to say here, however, that the MOEA signalis a triplicate signal MOE₁₋₁, MOE₁₋₂, MOE₁₋₃ and the MGA signal is alsoa triplicate signal MGE_(IA), MGE_(ILA), and MGE_(Il1A).

The MOEA and MREA signals from the logic section 5010 ₁ and the MOEB andMREB signals from the logic section 5010 ₂ are fed to a watch dog (WD)section 6006, to be described in more detail in connection with FIGS.15A, 15B, 15C, 15D and 15E. Suffice it to say here, however, that, asnoted above, the arbitration algorithm is a function of theoperating/non-operating condition of the logic sections 5010 ₁, 5010 ₂.This operating/non-operating condition is determined by the watchdogsection 6006 and more particularly by examining the MOEA, MREA, MOEB,MREB signals produced by the logic sections 5010 ₁ and 5010 ₂, 6002B,respectively. The MOEA, MREA, MOEB, MREB signals are asserted when thereis a grant. Such signals MOEA, MREA, MOEB, MREB are fed to the watchdogsection 6006. As will be described, the watchdog section 6006 examinesthe time history of these signals to determine if the logic section 5010₁ or 5010 ₂ asserting them is operating properly. Based on the resultsof such examination, the watchdog selects the Condition I, Condition II,or Condition III, described above.

More particularly, consider, for example, a case where the MOEA signalis asserted for too long a predetermined time interval. It should berecalled that the logic section 5010 ₁ producing such MOEA signal isgranted access to the memory in State 1 of the normal arbitrationcondition (i.e., Condition I, above). The watchdog section 6006 thusdetects a fault in logic section 5010 ₁. When such a fault is detected,the watchdog section 6006 issues a Condition III signal on in triplicateon lines MSAB to the arbitration sections 6004 ₁, 6004 ₂ in both thelogic sections 5010 ₁, 5010 ₂, respectively, indicating that thearbitration algorithm will operate in accordance with the States setforth above for Condition III. Further, the watchdog 6006 issues a dataoutput enable signal in triplicate on lines DOEA (i.e., DOEA₀, DOEA₁,and DOEA₂). This triplicate signal DOEA (i.e., DOEA₀, DOEA₁, and DOEA₂)is fed to a majority gate (MG) 6007 (FIGS. 13A, 13B, 13C, 13D and 13E),in accordance with the majority of the triplicate data fed to it,provides an enable/disable signal for gate 6009. If the majorityindicates a fault, the gate 6009 inhibits DATA from passing between thelogic section 5010 ₁ and the data port D thereof.

Consider the case where the arbitration is in Condition I. Consider alsothat in such condition I, the MREA signal is not produced after apredetermined time interval which ensures proper refreshing on theSDRAMs in the memory array region R₁. The watchdog section 6006 willagain detect a fault in the logic section 5010 ₁ port A controller6002A. When such a fault is detected, the watchdog section 6006 issues aCondition III signal on in triplicate on lines MSAB (i.e., MSAB₀, MSAB₁,MSAB₂) to the arbitration sections 6004 ₁, 6004 ₂ in both the logicsections 5010 ₁, 5010 ₂, respectively. Further, the watchdog 6006 issuesa data output enable signal in triplicate on lines DOEA (i.e., DOEA₀,DOEA₁, and DOEA₂) (FIGS. 13A, 13B, 13C, 13D and 13E) to inhibit DATAfrom passing between the logic section 5010 ₁ and the data port Dthereof.

Consider, for example, a case where the arbitration is in Condition Iand the MOEB signal from the logic section 5010 ₂ is asserted for toolong a predetermined time interval. The watchdog section 6006 thusdetects a fault in the logic section 5010 ₂. When such a fault isdetected, the watchdog section 6006 issues a Condition II signal on lineMSAB to the arbitration sections 6004 ₁, 6004 ₂ in both the logicsections 5010 ₁, 5010 ₂. Further, the watchdog 6006 issues a data outputenable signal in triplicate on lines DOEB to inhibit DATA from passingbetween the logic section 5010 ₂ and the data port D thereof.

It should be noted that the algorithm allows a transition betweenCondition II and Condition IV (i.e., reset) or from Condition III andCondition IV.

Thus, the arbitration logics 6004 ₁ and 6004 ₂ are adapted to issue thefollowing signals:

GRANT A (GA)-grant port A controller 6002B access to the memory arrayregion R₁;

GRANT B (GB)-grant port B controller 6002B access to the memory arrayregion R₁

REFRESH GRANT (GR)-grant the memory refresh section 6002R of logicsection 5010 ₁ access to the memory array region R₁ in Condition I andII or grant the memory refresh section 6002R of logic section 5010 ₂access to the memory array region R₁ in Condition III.

It should be noted that the details of GA and the other signal GB and GRare shown in more detail in connection with FIG. 19.

Thus, referring to FIGS. 13A, 13B, 13C, 13D and 13E, the memory arrayregion R₁ may be coupled to either Port_A (A) or Port_B (B) of the logicsections 5010 ₁, 5010 ₂ or to the memory refresh section 6002R thereinselectively in accordance with a Port_A_SELECT, Port_B_SELECT,Port_R_SELECT signal fed to a pair of selectors 6010 _(C), 6010 _(D),shown in more detail for exemplary logic section 5010 ₁. Access by theupper port A controller 6002A (i.e., Port_A), by the upper port Bcontroller 6002B, or the memory refresh section 6002R to the memoryarray region R₁ is in accordance with the algorithm described above.

An exemplary one of the upper port A and port B logic controllers 6002Aand 6002B, here controller 6002A, will be described in more detail inconnection with FIGS. 14A, 14B, 14C and 14D. Suffice it to say here,however, that it is noted that the output of selector ⁶⁰¹⁰ _(C) iscoupled to the control port C of the exemplary logic section 5101 ₁ andthe output of selector 6010 _(D) is coupled to the data port D of theexemplary logic section 5101 ₁ through the gate 6009. Each one of theselectors 6010 _(C) and 6010 _(D) has three inputs A, B, and R, asshown. The A, B and R inputs of selector 6010 _(C) are coupled to: theADR/CONTROL produced at the output of upper port A controller 6002A; theADR/CONTROL produced at the output of upper port B controller 6002B;and, the portion REFRESH_C of the refresh signal produced by the memoryrefresh section 6002R, respectively as indicated. The A, B and R inputsof selector 6010D are coupled to: the WRITE DATA produced at the outputof upper port A controller 6002A; the WRITE DATA produced at the outputof upper port B controller 6002B; and, the portion REFRESH_D of therefresh signal produced by the memory refresh section 6002R,respectively as indicated. The Port_A_SELECT, Port_B_SELECT are producedby the upper port A controller 6002A, upper port B controller 6002B in amanner to be described. The Port_R_SELECT signal is produced by thememory refresh section 6002R in a manner to be described to enableproper operation of the above described arbitration algorithm and toproper a refresh signal to the SDRAMs in the memory array region R₁ atthe proper time. Suffice it to say here, however, that when port Acontroller 6002A produces the Port_A_SELECT signal, the ADR/CONTROL atthe output of port A controller 6002A passes to the output of theselector 6010C and the DATA_WRITE at the output of the port A controller6002A passes to the output of the selector 6010D. Likewise, when port Bcontroller 6002B produces the Port_B_SELECT signal, the ADR/CONTROL atthe output of port B controller 6002B passes to the output of theselector 6010C and the DATA_WRITE at the output of the port B controller6002B passes to the output of the selector 6010D. In like manner, whenrefresh memory section 6002R produces the Port_R_SELECT_C signal, theREFRESH_C at the output of refresh memory section 8002R passes to theoutput of the selector 6010C and in response to the Port_R_SELECTsignal, the REFRESH_D at the output of the refresh memory section 8002Rpasses to the output of the selector 6010D.

It is noted that data read from the memory array R₁ (i.e., READ_DATA) isfed from the data port D to both the upper Port A controller 6002A andthe upper Port B controller 6002B.

Referring now to FIGS. 14A, 14B, 14C and 14D, the exemplary port Acontroller 6002A is shown in more detail to include a Port A primarycontrol section 6100P and a Port A secondary control section 6100S. Thetwo sections 6100P and 6100S are both coupled to port A and bothimplement the identical control logic. Thus, each one of the twosections 6100P and 6100S should produce the same results unless there isan error, here a hardware fault, in one of the two sections 6100P and6100S. Such a fault is detected by a fault detector 6102 in a manner tobe described.

Thus, referring to the details of one of the two sections 6100P and6100S, here section 6100P, it is first noted that the information atPort_A is fed to a parity checker 6101. It is noted that is there is anerror in parity induced by the CRC check described in FIGS. 12A, 12B,12C and 12D in connection with selector 5152, such detected parity erroris reported to a control and DATA path logic 6112. In response to adetected parity error, control and DATA path logic 6112 prevents memorycontrol signals (e.g., suppress the Column Address Select signal to theSDRAMs) from being produced on the CONTROL_P line. Thus, absent controlsignal, DATA will not be stored in the memory region.

The information at Port_A is also fed to a control register 6104 forstoring the memory control portion of the information at port A, an ADDRregister 6106 for storing the address portion (ADDR) of the informationat port A, a write data register 6108 (here a FIFO) for storing the DATAportion of the information at port A, such being the data which is to bewritten into the memory array region R₁. The control portion stored inregister 6104 is fed also to the control and data path logic 6112. Suchlogic 6112 produces: a memory array region requestRequest_Port_A_Primary (RAP) signal when the control portion in register6104 indicates that there is data to be stored in the memory arrayregion R₁; a Port A Primary Select (Port_A_P_SELECT) signal when thegrant has been issued thereto via a Grant_A_P signal (GAP) produced bythe arbitration logic 6004 ₁; and passes the control portion (CONTROL_P)stored in register 6104 to the output of the upper port A controller6002A, as indicated. It should be noted that the port A secondarycontrol section 6100S being fed the same information as the primarycontroller 6100P should produce the Same signals: here indicated as amemory array region request Request_Port A_SECONDARY (RAS) signal whenthe control portion in register 6104 indicates that there is data to bestored in the memory array region R₁; a Port A Secondary Select(Port_A_S_SELECT) signal when the grant has been issued thereto via aGrant_A_S signal (GAS) produced by the arbitration logic 6004 ₁.

The address portion stored in the ADDR register 6106 (ADDR_P) iscombined with the address portion ADDR_P stored in register 6106. BothCONTOL_P and ADDR_P are fed to a parity generator 6109 to produceADDR/CONTROL_P (which has both a content portion (n) and parity (p). Thecontent portion (n) of ADDR/CONTROL_P is led to a parity generator 6120to generate byte parity (p′) from the content portion (n) ofADDR/CONTROL_P. The generated parity (p′) is inverted by inverter 6122and the inverted parity is fed to a first input I₁ of the selector 6124.The content portion (n) of ADDR/CONRTOL_P is combined with a parity (p)produced at the output of selector 6124 in a manner to be described. Theparity (p) of ADDR/CONTROL_P is fed to a second input I₂ of the selector6124 and such parity (p) is also fed to an exclusive OR gate 6130. Alsofed to the exclusive OR gate 6130 is the parity (p) of the equivalentADDR/CONTROL_S signal produced by the Port A secondary control section6100S. As noted above, since both sections 600P and 600S are fed thesame information and implement the same logic functions, ADDR/CONTROL_Pshould be the same as ADDR/CONTROL_S unless there is a hardware fault inone of the sections 6100P, 6100S. If there is a fault (i.e., ifADDR/CONTROL_S and ADDR/CONTROL_P are different), the exclusive OR gate6130 will produce a logic 1 and in the absence of a fault, (i.e.,ADDR/CONTROL_S is the same as ADDR/CONTROL_P), the exclusive OR gate6130 will produce a logic 0.

In like manner, the content (n) of ADDR/CONTROL_P is fed to an exclusiveOR gate 6128. Also fed to the exclusive OR gate 6128 is the content (n)of the equivalent ADDR/CONTROL_S signal produced by the Port A secondarycontrol section 6100S. As noted above, since both sections 600P and 600Sare fed the same information and implement the same logic functions,ADDR/CONTROL_P should be the same as ADDR/CONTROL_S unless there is ahardware fault in one of the sections 6100P, 6100S. If there is a fault(i.e., if ADDR/CONTROL_S and ADDR/CONTROL_P are different), theexclusive OR gate 6128 will produce a logic 1 and in the absence of afault, (i.e., ADDR/CONTROL_S is the same as ADDR/CONTROL_P), theexclusive OR gate 6128 will produce a logic 0.

The outputs of exclusive OR gates 6128 and 6130 are fed to an OR gate6126. Thus, if there is an error in either the content (n) or the parity(p), the OR gate produces a logic 1; otherwise it produces a logic 0.The output of OR gate 6126 is fed to a fault detector 6102 which detectssuch a fault and reports such detected fault to the director. The Outputof OR gate 6126 is also fed as a control signal to selector 6124. If theOR gate produces a logic 1 (i.e., there is a fault), the selectorcouples the inverted parity of input I₁ to the output of selector 6124.This inverted parity is appended to the content (n) of ADDR/CONTROL_P tothereby corrupt such information. This corrupted information is detectedby the memory array region and converted into a “no-operation” commandas described in the above-referenced U.S. Pat. No. 5,943,287. On theother hand, if the OR gate 6126 produces a logic 0 (i.e., no fault), thenon-inverted parity at input I₂ of selector 6124 passes through selector6124 and is appended to the content portion (n) of ADDR/CONTROL/P.

A similar check is made with the DATA to be written into the memoryarray region. Thus, the DATA in register 6108 of primary controller6100P (WRITE_DATA_P) is fed to an exclusive OR gate 6116 along with thewrite DATA in the secondary controller 6100S (WRITE_DATA_S). (It isnoted the data in the write register 6108 of the primary controller6100P (DATA_WRITE_P) is fed to output DATA_WRITE bus while the writedata in the secondary controller 6100S (DATA_WRITE_S) is fed only to theexclusive OR gate 6118.) Thus, the exclusive OR gate 6116 produces alogic 0 if WRITE_DATA_P and WRITE_DATA_S are the same and produces alogic 1 if they are different. The fault detector 6102 detects suchlogic 1 and reports the detected fault to the transfer requestingdirector.

In like manner, a check is made of the DATA read (READ_DATA) from thememory array region R₁ which becomes stored in Read data register 6119,here a FIFO. The READ_DATA is fed to a read data register (here a FIFO)for transmission to the director via Port_A. Such READ_DATA in register6119 indicated as READ_DATA_P is fed to an exclusive OR gate 6118. Inlike manner, secondary controller 6100S should produce the same signalson output READ_DATA_S. READ_DATA_P and READ_DATA_S are fed to anexclusive OR gate 6118. Thus, the exclusive OR gate 6118 produces alogic 0 if READ_DATA_P and READ_DATA_S are the same and produces a logic1 if the are different. The fault detector 6102 detects such logic 1 andreports the detected fault to the transfer requesting director.

It is noted that the RAP and PAS signals are both sent to thearbitration logic 6004 ₁ (FIGS. 13A, 13B, 13C, 13D and 13E) as compositesignal REQUEST A. The arbitration section 6004 ₁ considers a validrequest only if both signals RAP and RAS are the same. In like manner,the arbitration logic 6004 ₁ issues separate grant signals GAP and GASwhich are shown in FIGS. 13A, 13B, 13C, 13D and 13E as a compositesignal GRANT_A. Likewise, PORT_A_P_SELECT and PORT_A_S_SELECT signalsare both sent to the arbitration logic 6004 ₁ (FIGS. 13A, 13B, 13C, 13Dand 13E) as composite signal PORT_A_SELECT. The arbitration section 6004₁ considers a valid request only if both signals PORT_A_P_SELECT andPORTA_S_SELECT are the same.

As noted above, the upper port B controller 6002B provides signals: RBP,GBP, PORT_B_P_SELECT, ADDR/CONTROL, DATA_WRITE RBS, GBS, PORT B_SELECT,and READ_DATA, which are equivalent to RAP, GAP, PORT A_SELECT,ADR/CONTROL, DATA_WRITE, RAS, GAS, PORT A_SELECT, and READ_DATA,respectively, which are provided by the upper port A controller 6002A.

Referring now to FIGS. 15A, 15B, 15C, 15D and 15E, the arbitrationlogics 6004 ₁, 6004 ₂ of the logic sections 5010 ₁, 5010 ₂,respectively, are shown along with the watchdog section 6006. It isfirst noted that the arbitration logic 6004 ₁, 6004 ₂ are identical inconstruction.

Arbitration logic 6004 ₁ is fed by:

REQUEST A (i.e., RAP, RAS) from upper port A controller 6002A of logicsection 5010 ₁ (FIGS. 13A, 13B, 13C, 13D and 13E);

REQUEST B (RBP, RBS) from upper port B controller 6002B of logic section5010 ₁ (FIGS. 13A, 13B, 13C, 13D and 13E);

REQUEST R from upper memory refresh section 6002R of logic section 5010₁ (FIGS. 13A, 13B, 13C, 13D and 13E) (It is to be noted that the REQUESTR is made up of two signals, each being produced by identical primaryand secondary identical memory refresh units, not shown, in memoryrefresh section 6002R both of which have to produce the same refreshsignal in order for the arbitration logic 6004 ₁ to respond to therefresh request). Arbitration logic 6004 ₂ is fed by:

REQUEST A from upper port A controller 6002A of logic section 5010 ₂(FIGS. 13A, 13B, 13C, 13D and 13E);

REQUEST B from upper port B controller 6002B of logic section 5010 ₂(FIGS. 13A, 13B, 13C, 13D and 13E);

REQUEST R from upper memory refresh section 6002R of logic section 5010₂.

As shown in FIGS. 15A, 15B, 15C, 15D and 15E, each one of the threerequest signals REQUEST A, REQUEST B, and REQUEST R, produced in logicsection 5010 ₁ is fed, in triplicate, to three identical arbitrationunits, (i.e., arbitration unit I, arbitration unit II, and arbitrationunit III) in the arbitration logic 6004 ₁ of such logic section 5010 ₁,as indicated. (See also FIG. 19). Likewise, each one of the threerequest signals REQUEST A, REQUEST B, and REQUEST R, produced in logicsection 5010 ₂ is fed, in triplicate, to three identical arbitrationunits, (i.e., arbitration unit I, arbitration unit II, and arbitrationunit III, in the arbitration logic 6004 ₂ of such logic section 5010 ₂as indicated.

In response to such request signals, REQUEST A, REQUEST B, and REQUESTR, each arbitration unit I, II, and III determines from the threerequests; i.e., REQUEST A, REQUEST B, and REQUEST R, fed to it and inaccordance with the algorithm described above, whether upper port Acontroller 6002A, upper port B controller 6002B, or the memory refresh6002R is to be given access to the memory array region R₁. As notedabove, the operating Condition (i.e., Condition I, Condition II, orCondition III) is a function of whether the logic section 5010 ₁ isoperating properly and whether the logic section 5010 ₂ is operatingproperly. The watchdog section 2006 determines whether such logicsections 5010 ₁, 5010 ₂ are operating properly. More particularly, whenthe arbitration units I, II, and III make their decision, they alsoproduce a memory output enable (MOE) signals MOEI, MOEII and MOEIII,respectively, (when either logic section 5010 ₁ or 5010 ₂ is to begranted access to the memory array region R₁) and a memory refreshsignal MREs (i.e., MREI, MREII and MREIII, respectively, when memoryrefresh section 6002R is to be granted access to the memory array regionR₁). Thus, MOE signals MOEI₁, MOEII₁, and MOEIII₁ are produced byarbitration units I, II, and III, respectively, in arbitration logic6004 ₁. Also, MRE signals MREI₁, MREII₁, and MREIII₁ are produced byarbitration units I, II, and III, respectively, in arbitration logic6004 ₁. In like manner, MOE signals MOEI₂, MOEII₂, and MOEIII₂ areproduced by arbitration units I, II, and III, respectively, inarbitration logic 6004 ₂. Also, MRE signals MREI₂, MREII₂, and MREIII₂are produced by arbitration units I, II, and III, respectively, inarbitration logic 6004 ₂. (See also FIG. 19).

These signals are fed to each of three identical watchdogs, WD_(I),WD_(II), WD_(III) as follows:

The MOE and MRE signals produced by the arbitration unit I inarbitration logics 6004 ₁ and 6004 ₂ (i.e., MOEI₁, MOEI₂, MREI₁ andMREI₂) are fed to watchdog WD_(I);

The MOE and MRE signals produced by the arbitration unit II inarbitration logics 6004 ₁ and 6004 ₂ (i.e., MOEII₁, MOEII₂, MREII₁ andMREII₂) are fed to watchdog WD_(II); and

The MOE and MRE signals produced by the arbitration unit III inarbitration logics 6004 ₁ and 6004 ₂ (i.e., MOEIII₁, MOEIII₂, MREIII₁and MREIII₂) are fed to watchdog WD_(III).

Each one of the watchdogs I, II, III is implemented and arrangedidentical to perform the same logic functions; however, they preferablyimplemented with components manufactured independently of each other.Further, each one of the watchdogs I, II, and III operates ill responseto its own independent clock, i.e., Clock I, Clock II, and Clock III,respectively. Thus, each watchdog makes an independent determination asto whether these signals are in proper time and rate and thus,determine, in accordance with the “Condition algorithm” described above,the proper one of the Conditions (i.e., Condition I, Condition II, orCondition III) for the system. An indication of the Condition isprovided by each of the watchdogs WD_(I), WD_(II), and WD_(III) as atwo-bit word MSAB_(I), MSAB_(II), and MSAB_(III), respectively. Thetwo-bit word is produces as follows:

00=Condition I

01=Condition II

10=condition III

11=Reset (i.e., Condition IV)

These three words MSAB_(I), MSAB_(II), and MSAB_(III) are fed to botharbitration logics 6004 ₁ and 6004 ₂, as indicated. It should beremembered that each one of the arbitration logics 6004 ₁ and 6004 ₂(and hence the arbitration logics 6004 ₁ and 6004 ₂ therein), operatewith a separate independent clock, Clock 1, and Clock 2, respectively.In order to synchronize the three words MSAB_(I), MSAB_(II), andMSAB_(III) are fed to logic section 5010 ₁ and fed to logic section 5010₂. Each one of the arbitration logics 6004 ₁, 6004 ₂ has asynchronization filter 6200 ₁, 6200 ₂ to be described. Suffice it to sayhere, however, that the filter 6200 ₁ produces corresponding signalsMSAB_(I) _(—) ₁, MSAB_(II) _(—) ₁, and MSAB_(III) _(—) ₁, respectively,and filter 6200 ₂ produce corresponding signals MSAB_(I) _(—) ₂,MSAB_(II) _(—) ₂, and MSAB_(III) _(—) ₂, respectively, as indicated.

The signals MSAB_(I) _(—) ₁, MSAB_(II) _(—) ₁, and MSAB_(III) _(—) ₁,are fed to the arbitration units I, II, and III, respectively, inarbitration logic 6004 ₁. In like manner, the signals MSAB_(I) _(—) ₂,MSAB_(II) _(—) ₂, and MSAB_(III) _(—) ₂, are fed to the arbitrationunits I, II, and III, respectively, in arbitration logic 6004 ₂. Inresponse to such signals, each one of the arbitration units I, II, andIII, makes an independent determination of whether logic section 5010 ₁(FIGS. 13A, 13B, 13C, 13D and 13E) or logic section 5010 ₂ will begranted access to the memory array region R₁. A grant by logic section5010 ₁ to logic section 5010 ₂ is indicated by a Memory Grant (MG)signal. Thus, arbitration units I, II, and III of logic section 5010 ₁produce Memory Grant signals MGI₁, MGII₁, and MGIII₁, respectively. Suchsignals are fed to a synchronization filter 6202 ₂ in arbitration logic6004 ₂. The synchronization filter 6202 ₂ operates as is constructed inthe same manner as synchronization filters 6200 ₁ and 6200 ₂. In likemanner arbitration units I, II, and III of logic section 5010 ₂ produceMemory Grant signals MGI₂, MGII₂, and MGIII₂, respectively. Such signalsare fed to a synchronization filter 6202 ₁ in arbitration logic 6004 ₁.The synchronization filter 6202 ₁ operates as is constructed in the samemanner as synchronization filter 6202 ₂.

Thus, considering exemplary synchronization filter 6202 ₂, such filteris fed by the three Memory Grant (MG) signals MGI₂, MGII₂, and MGIII₂,as indicated. The three signals are stored in registers 6204I, 6204IIand 6204III, respectively, in response to a clock pulse produced by theClock 2. Each of the three registers 6204I, 6204II and 6204III, send theinformation stored therein to each of three majority gates MGI, MGII,and MGIII, respectively, as indicated. The majority gates produce anoutput which is the majority of the three inputs fed thereto. Theoutputs of the three majority gates MGI, MGII and MGIII are thearbitration units I, II and III, respectively, in the arbitration logic6004 ₂, as indicated.

More particularly, referring to FIG. 16, portions of arbitration logics6004 ₁ and 6004 ₂ are shown. The data to be fed to the output ofarbitration logic 6004 ₁ is clocked into register 7000 ₁ of arbitrationI, register 7000 ₂ of arbitration II, and register 7000 ₃ of arbitrationIII simultaneously in response to the same clock pulse produced by Clock1. Thus, each of the registers 7000 ₁, 7000 ₂, 7000 ₃ should store thesame data at the clock pulse produced by Clock 1, as indicated in FIG.18. The data is then fed to registers 7002 ₁, 7002 ₂, 7002 ₃ of filter6202 ₂ of arbitration logic 6004 ₂. The data at the registers 7002 ₁,7002 ₂, 7002 ₃ are stored therein in response to the same clock producedby Clock 2. Because of the data in registers 7000 ₁, 7000 ₂, 7000 ₃arrive at registers 7002 ₁, 7002 ₂, 7002 ₃ with different delays asindicated in FIG. 18, while the data in 7000 ₁, 7000 ₂ 7000 ₃ is thesame, here the data stored in registers 7002 ₁, 7002 ₂, 7002 ₃ may bedifferent as shown in FIG. 18. The data stored in register 7002 ₁ is fedto majority gates (MGs) 7004 ₁, 7004 ₂ and 7004 ₃. The data stored inregister 7002 ₂ is also fed to majority gates (MGs) 7004 ₁, 7004 ₂ and7004 ₃. Likewise, the data stored in register 7002 ₃ is fed to majoritygates (MGs) 7004 ₁, 7004 ₂ and 7004 ₃. Each one of the majority gatesMGs produces an output representative of the majority of the logicsignals fed thereto as indicated in FIG. 17.

Referring now to FIGS. 20A, 20B and 20C, the three arbitrations I, II,and III of exemplary arbitration logic 6004 ₁ are the signals fedthereto and produced thereby are shown in more detail. It is first notedthat the primary signal REQUEST_A_P, (RAP), and the secondary requestsignal REQUEST_A_S (RAS) are each fed in triplicate; one copy to each ofthe arbitrations I, II, and III, as indicated. The one of the triplicateRAP and RAS fed to arbitration I are fed to an AND gate 8000 ₁, a secondone of the triplicate RAP and RAS fed to arbitration II are fed to anAND gate 8000 ₂, and the third one of the triplicate RAP and RAS fed toarbitration III are fed to an AND gate 8000 ₃, as indicated. Likewise,the signals REQUEST_B_P, (RBP), and REQUEST_B_S (RBS) are each fed intriplicate; one copy to each of the arbitrations I, II, and III, asindicated. The one of the triplicate RBP and RBS fed to arbitration Iare fed to an AND gate 8002 ₁, a second one of the triplicate RBP andRBS fed to arbitration II are fed to an AND gate 8002 ₂, and the thirdone of the triplicate RBP and RBS fed to arbitration III are fed to anAND gate 8002 ₃, as indicated. As mentioned briefly above, there are twomemory refresh units in the memory refresh section 6002R(FIGS. 13A, 13B,13C, 13D and 13E). One, a primary unit (not shown), issues a request RRPand the other, a secondary unit (not shown), issues a request RRS.Above, in connection with FIGS. 13A, 13B, 13C, 13D and 13E, these tworequests were considered as a composite request (REFRESH_REQUEST) tosimplify the discussion presented above. Here, in connection with FIG.19, the individual signals RRP, RRS are shown in more detail. Thus, thesignals RRP, RRS are each fed in triplicate; one copy to each of thearbitrations I, II, and III, as indicated. The one of the triplicate RRPand RRS is fed to arbitration I are fed to an AND gate 8004 ₁, a secondone of the triplicate RRP and RRS fed to arbitration II are fed to anAND gate 8004 ₂, and the third one of the triplicate RRP and RS fed toarbitration III are fed to an AND gate 8004 ₃, as indicated.

Thus, in the case of each pair, in order for the request to be issued tothe arbitration I, II, or III, the AND gate associated therewith mustsee the same request from both the primary signal and the secondarysignal fed to it.

Each arbitration I, II and II issues pairs of grants, i.e., a primarygrant to the primary unit and a secondary grant to the secondary unit.Thus, each of the arbitrations I, II and III issues: the primary andsecondary grants (GAP and GAS, respectively) to the Port A primarycontrol section 6100P (FIGS. 14A, 14B, 14C and 14D) and Port A secondarycontrol section 6100S of Port A controller 6002A; the primary andsecondary grants (GBP and GBS, respectively) to the Port B primarycontrol section and Port A secondary control section of Port Bcontroller 6002B; and the primary and secondary grants (GRP and GRS,respectively) to the memory refresh primary unit memory refreshsecondary unit of the memory refresh section 6002R (FIGS. 13A, 13B, 13C,13D and 13E).

The arbitrations I, II, and III produce Memory Output Enable signalsMOE_(I-1), MOE_(II-1), and MOE_(III-1) respectively, as indicated, forthe watchdogs WD_(I), WD_(II), and WD_(II), respectively, as shown inFIGS. 15A, 15B, 15C, 15D and 15E. The arbitrations I, II, and IIIproduce Memory Refresh Enable signals MRE_(I-1), MRE_(II-1), andMRE_(III-1), respectively, as indicated, for the watchdogs WD_(I),WD_(II) and WD_(III), respectively, as shown in FIGS. 15A, 15B, 15C, 15Dand 15E. The arbitrations I, II, and III produce Memory Grant signalsMG_(I), MG_(I), and MG_(III), respectively, as indicated, for theregisters 6204 _(I), 6204 _(II), and 6204 _(III), respectively, offilter 6202 ₂ of logic section 5010 ₂, as shown in FIGS. 15A, 15B, 15C,15D and 15E.

Thus, it should be noted that while each one of the registers 7002 ₁,7002 ₂, 7002 ₃, of filters 6002 ₁, 6002 ₂ (FIGS. 19), are fed the samedata from registers 7000 ₁, 7000 ₂, and 7000 ₃, respectively, because ofthe time skew shown in FIG. 18, such registers 7002 ₁, 7002 ₂, 7002 ₃,may not store the data which is in registers 7000 ₁, 7000 ₂, and 7000 ₃,respectively. However, the majority gates MG 7004 ₁-7004 ₃ will producethe same data according to FIG. 17. Therefore, the three arbitrations I,II, and III of arbitration logic 6004 ₂ will receive the same data(i.e., the data produced by the majority gates MG 7004 ₁-7004 ₃) therebyproviding coherency (i.e., synchronization) to tile arbitrations I, II,and III even though the arbitrations are operating independently of eachother.

Other embodiments are within the spirit and scope of the appendedclaims.

What is claimed is:
 1. A memory system, comprising: a backplane having a plurality of receiving slots, each one of the slots having electrical contacts for providing an indication of such one of the slots, each one of the slots having a different slot indication; a plurality of memory boards, each one being plugged into a corresponding one of the slots, each one of such boards being coupled to the electrical contacts in the corresponding one of the slots to provide a slot signal indicative of the slot indication provided by the electrical contacts; each one of such boards having: a memory array region; and a switching network for transferring information between a port of the switching network and a memory on such memory board, such transfer being initiated by a director coupled to such port, such director designating a selected one of the plurality of memory boards, such director providing to the switching network a “tag” indicating such designated one of one of the plurality of memory boards having the memory involved in the requested transfer, such switch network having: a memory board checker, for comparing the slot signal with the “tag” for indicating whether the memory board receiving the “tag” is the director designated one of the plurality of memory boards.
 2. The memory system recited in claim 1 wherein the memory board checker sends an indication to the requesting director slot signal as to whether the memory board receiving the “tag” is the designated one of the memory boards.
 3. The memory system recited in claim 1 wherein the switching network includes a decoder responsive to the indication provided by the memory board checker for inhibiting transfer of the information at the port to the memory if the memory board receiving the “tag” is different from the designated one of the memory boards.
 4. The memory system recited in claim 2 wherein the switching network includes a decoder responsive to the indication provided by, the memory board checker for inhibiting transfer of the information at the port to the memory if the memory board receiving the “tag” is different from the designated one of the memory boards.
 5. A memory system, comprising: a backplane having a plurality of receiving slots, each one of the slots having electrical contacts for providing an indication of such one of the slots, each one of the slots having a different slot indication; a plurality of memory boards, each one being plugged into a corresponding one of the slots, each one of such boards being coupled to the electrical contacts in the corresponding one of the slots to provide a slot signal indicative of the slot indication provided by the electrical contacts; each one of such boards having: (A) a plurality of memory array regions, each one of such regions having a common DATA port and a pair of independent memory array control ports; (B) a plurality of logic sections, each one of the logic sections having: a first logic section upper ports, a second logic section upper ports, a logic section control port, and a logic section DATA port; (C) a plurality of crossbar switches, each one thereof having a plurality of crossbar switch upper ports and a plurality of crossbar switch lower ports, such crossbar switch upper ports and crossbar switch lower ports being coupled in accordance with information fed to the crossbar switch upper ports, such lower ports being coupled to the first and second upper ports of the plurality of logic sections, for transferring information between a selected one of the upper ports of the crossbar switches and the control logic sections to one of the memory array regions coupled thereto, such transfer being initiated by a director coupled to such selected one of the upper ports, such director designating a selected one of the plurality of memory boards, such director providing to the switching network a “tag” indicating such designated one of one of the plurality of memory boards having the memory involved in the requested transfer, such crossbar switches having: a memory board checker, for comparing the slot signal with the “tag” for indicating(whether the memory board receiving the “tag” is the director designated one of the plurality of memory boards.
 6. The memory system received in claim 5 wherein the memory board checker sends an indication to the requesting director slot signal as to whether the memory board receiving the “tag” is the designated one of the memory boards.
 7. The memory'system recited in claim 5 wherein the switching network includes a decoder responsive to the indication provided by the memory board checker for inhibiting transfer of the information at the port to the memory if the memory board receiving the “tag” is different from the designated one of the memory boards.
 8. The memory system recited in claim 6 wherein the switching network includes a decoder responsive to the indication provided by the memory board checker for inhibiting transfer of the information at the port to the memory if the memory board receiving, the “tag” is different from the designated one of the memory boards.
 9. The memory system recited in claim 5 wherein: the plurality of logic sections are arranged in pairs, each pair being coupled to a corresponding one of the plurality of array regions; the logic section in each pair thereof having: the DATA port of such pair of logic sections connected to the memory DATA port of the corresponding one of the memory array region; the control port thereof connected to a corresponding one of the pair of memory array control ports the corresponding one of the memory array regions; and wherein each one of the plurality of crossbar switches comprises a plurality of crossbar switch upper ports and a plurality of crossbar switch lower ports, such crossbar switch upper ports and crossbar switch lower ports being coupled in accordance with information fed to the crossbar switch upper ports, wherein a first one of the crossbar switches has each one of the crossbar switch lower ports thereof connected to the first logic section upper port of a first one of the logic sections in a corresponding one of the pairs of such logic sections wherein a second one of the crossbar switches has each one of the crossbar switch lower ports thereof connected to the second logic section upper port of the first one of the logic sections in a corresponding one of the pairs of such logic sections; wherein a third one of the crossbar switches has each one of the crossbar switch lower ports thereof connected to the first logic section upper port in a second one of the logic sections in a corresponding one of the pairs of such logic sections; and wherein a fourth one of the crossbar switches has each one of the crossbar switch lower ports thereof connected to the second one of the pair of logic section upper ports of a corresponding one of the second logic sections in a corresponding one of the pairs of such logic sections.
 10. The memory system received in claim 9 wherein the memory board checker sends an indication to the requesting director slot signal as to whether the memory board receiving the “tag” is the designated one of the memory boards.
 11. The memory system recited in claim 9 wherein the switching network includes a decoder responsive to the indication provided by the memory board checker for inhibiting transfer of the information at the port to the memory if the memory board receiving the “tag” is different from the designated one of the memory boards.
 12. The memory system recited in claim 10 wherein the switching network includes a decoder responsive to the indication provided by the memory board checker for inhibiting transfer of the information at the port to the memory if the memory board receiving the “tag” is different from the designated one of the memory boards. 